Mutual Information

The Mutual Information I(X,Y)I(X, Y) measures the amount of information gained on variable XX (the reduction in the Expected Log-Loss) by observing variable YY:

I(X,Y)=H(X)โˆ’H(XโˆฃY)I(X,Y) = H(X) - H(X|Y)

The Venn Diagram below illustrates this concept:

The Conditional Entropy H(XโˆฃY)H(X|Y) measures, in bits, the Expected Log-Loss associated with variable XX once we have information on variable YY:

H(XโˆฃY)=โˆ’โˆ‘yโˆˆYp(y)โˆ‘xโˆˆXp(xโˆฃy)logโก2(p(xโˆฃy))H(X|Y) = - \sum\limits_{y \in Y} {p(y)\sum\limits_{x \in X} {p(x|y){{\log }_2}} } \left( {p(x|y)} \right)

Hence, the Conditional Entropy is a key element in defining the Mutual Information between XX and YY.

Note that

I(X,Y)=H(X)โˆ’H(XโˆฃY)I(X,Y) = H(X) - H(X|Y)

is equivalent to:

I(X,Y)=โˆ‘xโˆˆXโˆ‘yโˆˆYp(x,y)logโก2p(x,y)p(x)p(y)I(X,Y) = \sum\limits_{x \in X} {\sum\limits_{y \in Y} {p(x,y){{\log }_2}} } {{p(x,y)} \over {p(x)p(y)}}

and furthermore equivalent to:

I(X,Y)=โˆ‘yโˆˆYp(y)โˆ‘xโˆˆXp(xโˆฃy)logโก2p(xโˆฃy)p(x)I(X,Y) = \sum\limits_{y \in Y} {p(y)\sum\limits_{x \in X} {p(x|y){{\log }_2}} } {{p(x|y)} \over {p(x)}}

This allows computing the Mutual Information between any two variables.


For a given network, BayesiaLab can report the Mutual Information in several contexts:

  • Main Menu > Analysis > Report > Target > Relationship with Target Node.

  • Note that this table shows the Mutual Information of each node, e.g., XRay, Dyspnea, etc., only with regard to the Target Node, Cancer.

  • Main Menu > Analysis > Report > Relationship Analysis:

  • The Mutual Information can also be shown by selecting Main Menu > Analysis > Visual > Overall > Arc > Mutual Information and then clicking the Show Arc Comments icon or selecting Main Menu > View > Show Arc Comments.

  • Note that the corresponding options under Preferences > Analysis > Visual Analysis > Arc's Mutual Information Analysis have to be selected first:

  • In Preferences, Child refers to the Relative Mutual Information from the Parent onto the Child node, i.e., in the direction of the arc.

  • Conversely, Parent refers to the Relative Mutual Information from the Child onto the Parent node, i.e., in the opposite direction of the arc.

Last updated


Bayesia USA

Bayesia S.A.S.

Bayesia Singapore

Copyright ยฉ 2024 Bayesia S.A.S., Bayesia USA, LLC, and Bayesia Singapore Pte. Ltd. All Rights Reserved.