Skip to Content
BayesiaLabKey ConceptsMutual Information

Mutual Information

Mutual Information I(X,Y)I(X,Y) measures how much observing YY reduces uncertainty about XX, on average.

I(X,Y)=H(X)−H(X∣Y)I(X,Y) = H(X) - H(X \mid Y)

The Venn Diagram below illustrates this concept:

Loading SVG...

Conditional Entropy H(X∣Y)H(X \mid Y) is the uncertainty about XX after observing YY, averaged over the possible observations of YY. It is the Expected Log-Loss in bits when predicting XX from YY.

H(X∣Y)=−∑yp(y)∑xp(x∣y)log⁡2p(x∣y)H(X \mid Y) = -\sum_y p(y)\sum_x p(x\mid y)\log_2 p(x\mid y)

Mutual Information can also be computed from the joint probabilities or from the conditional probabilities.

I(X,Y)=∑x∑yp(x,y)log⁡2p(x,y)p(x)p(y)I(X,Y) = \sum_x \sum_y p(x,y)\log_2\frac{p(x,y)}{p(x)p(y)}I(X,Y)=∑yp(y)∑xp(x∣y)log⁡2p(x∣y)p(x)I(X,Y) = \sum_y p(y)\sum_x p(x\mid y)\log_2\frac{p(x\mid y)}{p(x)}

Usage

For a given network, BayesiaLab can report the Mutual Information in several contexts:

Target Analysis report showing Mutual Information for each nodeRelationship Analysis report showing Mutual Information
Loading SVG...
  • Note that the corresponding options under Menus > Window > Preferences > Analysis > Visual Analysis > Arc's Mutual Information Analysis have to be selected first:
Loading SVG...
  • In Preferences, Child refers to the Relative Mutual Information from the Parent onto the Child node, i.e., in the direction of the arc.
  • Conversely, Parent refers to the Relative Mutual Information from the Child onto the Parent node, i.e., in the opposite direction of the arc.