Expected Log-Loss

Context

The Log-Loss $LL(E)$ reflects the number of bits required to encode an n-dimensional piece of evidence (or observation) $E$ given the current Bayesian network $B$. As a shorthand for "the number of bits required to encode," we use the term "cost" in the sense that "more bits required" means computationally "more expensive."

$L{L_B}(E) = - {\log _2}\left( {{P_B}(E)} \right),$

where $PB(E)$ is the joint probability of the evidence $E$ computed by the network $B$:

Furthermore, one of the key metrics in Information Theory is Entropy:

$H(X) = - \sum\limits_{x \in X} {p(x){{\log }_2}\left( {p(x)} \right)}$

As a result, Entropy can be considered the sum of the Expected Log-Loss values of each state $x$ of variable $X$ given network $B$.

$H(X) = \sum\limits_{x \in X} {L{L_x}}$

where

$L{L_x} = - {p_B}(x){\log _2}\left( {{p_B}(x)} \right)$

Usage

To illustrate these concepts, we use the familiar Visit Asia network:

VisitAsia.xbl (opens in a new tab)

In BayesiaLab, the Expected Log-Loss values can be shown in the context of the Monitors.

Monitors

We consider the nodes $Dyspnea$ and $Bronchitis$ in the VisitAsia.xbl network.

• On the left, the Monitors of the two nodes show their marginal distributions.
• On the right, we set $p\left( {Bronchities = True} \right) = 100\%$, which updates the probability of $Dyspnea$, i.e., $p\left( {Dyspnea = True} \right) = 80.54\%$.

• On the Monitor of $Dyspnea$, we now select Monitor Context Menu > Show Expected Log-Loss that the Expected Log-Loss values for the states of $Dyspnea$ are shown instead of their probabilities.
• This is an interesting example because setting $Bronchitis=True$ does reduce the Entropy of $Dyspnea=True$, but does not seem to change the Expected Log-Loss of $Dyspnea=False$.

Visual Illustration

The following plot illustrates $H(Dyspnea)$, $LL(Dyspnea=True)$, and $LL(Dyspnea=False)$. For a compact representation in the plot, we substituted $X$ for $Dyspnea$.

In this binary case, the curves show how the $Entropy H(X)$ can be decomposed into $LL(x=True)$ and $LL(x=False)$.

The blue curve also confirms that the Expected Log-Loss values are identical for the two probabilities of $Dyspnea=False$, i.e., 80.54% and 42.52%.

• $LL(p(Dyspnea=False)=1-0.1946=0.8054)=0.459$
• $LL(p(Dyspnea=False)=1-0.5748=0.4252)=0.459$

Monitor Tooltip

Instead of replacing the states' probabilities with the Expected Log-Loss values in a Monitor, you can bring up the Expected Log-Loss values ad hoc as a Tooltip.

• Click on the Information Mode icon in the Toolbar.
• Then, when you hover over any Monitor with your cursor, a Tooltip shows the Expected Log-Loss values.

Workflow Animation