Mutual Information

The Mutual Information I(X,Y)I(X, Y) measures the amount of information gained on variable XX (the reduction in the Expected Log-Loss) by observing variable YY:

I(X,Y)=H(X)โˆ’H(XโˆฃY)I(X,Y) = H(X) - H(X|Y)

The Venn Diagram below illustrates this concept:

The Conditional Entropy H(XโˆฃY)H(X|Y) measures, in bits, the Expected Log-Loss associated with variable XX once we have information on variable YY:

H(XโˆฃY)=โˆ’โˆ‘yโˆˆYp(y)โˆ‘xโˆˆXp(xโˆฃy)logโก2(p(xโˆฃy))H(X|Y) = - \sum\limits_{y \in Y} {p(y)\sum\limits_{x \in X} {p(x|y){{\log }_2}} } \left( {p(x|y)} \right)

Hence, the Conditional Entropy is a key element in defining the Mutual Information between XX and YY.

Note that

I(X,Y)=H(X)โˆ’H(XโˆฃY)I(X,Y) = H(X) - H(X|Y)

is equivalent to:

I(X,Y)=โˆ‘xโˆˆXโˆ‘yโˆˆYp(x,y)logโก2p(x,y)p(x)p(y)I(X,Y) = \sum\limits_{x \in X} {\sum\limits_{y \in Y} {p(x,y){{\log }_2}} } {{p(x,y)} \over {p(x)p(y)}}

and furthermore equivalent to:

I(X,Y)=โˆ‘yโˆˆYp(y)โˆ‘xโˆˆXp(xโˆฃy)logโก2p(xโˆฃy)p(x)I(X,Y) = \sum\limits_{y \in Y} {p(y)\sum\limits_{x \in X} {p(x|y){{\log }_2}} } {{p(x|y)} \over {p(x)}}

This allows computing the Mutual Information between any two variables.

Usage

For a given network, BayesiaLab can report the Mutual Information in several contexts:

  • Main Menu > Analysis > Report > Target > Relationship with Target Node.

  • Note that this table shows the Mutual Information of each node, e.g., XRay, Dyspnea, etc., only with regard to the Target Node, Cancer.

  • Main Menu > Analysis > Report > Relationship Analysis:

  • The Mutual Information can also be shown by selecting Main Menu > Analysis > Visual > Overall > Arc > Mutual Information and then clicking the Show Arc Comments icon or selecting Main Menu > View > Show Arc Comments.

  • Note that the corresponding options under Preferences > Analysis > Visual Analysis > Arc's Mutual Information Analysis have to be selected first:

  • In Preferences, Child refers to the Relative Mutual Information from the Parent onto the Child node, i.e., in the direction of the arc.

  • Conversely, Parent refers to the Relative Mutual Information from the Child onto the Parent node, i.e., in the opposite direction of the arc.

Last updated

Logo

Bayesia USA

info@bayesia.us

Bayesia S.A.S.

info@bayesia.com

Bayesia Singapore

info@bayesia.com.sg

Copyright ยฉ 2024 Bayesia S.A.S., Bayesia USA, LLC, and Bayesia Singapore Pte. Ltd. All Rights Reserved.