Expected Log-Loss

Context

The Log-Loss $LL(E)$ reflects the number of bits required to encode an n-dimensional piece of evidence (or observation) $E$ given the current Bayesian network $B$. As a shorthand for “the number of bits required to encode,” we use the term “cost” in the sense that “more bits required” means computationally “more expensive.”

where $PB(E)$ is the joint probability of the evidence $E$ computed by the network $B$:

Furthermore, one of the key metrics in Information Theory is Entropy:

As a result, Entropy can be considered the sum of the Expected Log-Loss values of each state $x$ of variable $X$ given network $B$.

where

Usage

To illustrate these concepts, we use the familiar Visit Asia network:

In BayesiaLab, the Expected Log-Loss values can be shown in the context of the Monitors.

Monitors

We consider the nodes $\mathit{Dyspnea}$ and $\mathit{Bronchitis}$ in the VisitAsia.xbl network.

• On the left, the Monitors of the two nodes show their marginal distributions.
• On the right, we set $P(\mathit{Bronchities} =\mathrm{True}) = 100\%$, which updates the probability of $Dyspnea$, i.e., $P(\mathit{Dyspnea} = \mathrm{True}) = 80.54\%$.

• On the Monitor of $Dyspnea$, we now select Monitor Context Menu > Show Expected Log-Loss that the Expected Log-Loss values for the states of $Dyspnea$ are shown instead of their probabilities.
• This is an interesting example because setting $Bronchitis=True$ does reduce the Entropy of $Dyspnea=True$, but does not seem to change the Expected Log-Loss of $Dyspnea=False$.

Illustration

The following plot illustrates $H(Dyspnea)$, $LL(Dyspnea=True)$, and $LL(Dyspnea=False)$. For a compact representation in the plot, we substituted $X$ for $Dyspnea$.

In this binary case, the curves show how the $Entropy\ H(X)$ can be decomposed into $LL(x=\mathrm{True})$ and $LL(x=\mathrm{False})$.

The blue curve also confirms that the Expected Log-Loss values are identical for the two probabilities of $Dyspnea=False$, i.e., 80.54% and 42.52%.

• $LL\big(P(Dyspnea=\mathrm{False})=1-0.1946=0.8054\big)=0.459$
• $LL\big(P(Dyspnea=\mathrm{False})=1-0.5748=0.4252\big)=0.459$

Monitor Tooltip

Instead of replacing the states’ probabilities with the Expected Log-Loss values in a Monitor, you can bring up the Expected Log-Loss values ad hoc as a Tooltip.

• Click on the Information Mode icon in the Toolbar.
• Then, when you hover over any Monitor with your cursor, a Tooltip shows the Expected Log-Loss values.

Workflow Animation