Mutual Information

The Mutual Information I(X,Y)I(X, Y)I(X,Y)
 measures the amount of information gained on variable XXX
 (the reduction in the Expected Log-Loss) by observing variable YYY
:
​I(X,Y)=H(X)−H(X∣Y)I(X,Y) = H(X) - H(X|Y)I(X,Y)=H(X)−H(X∣Y)
​
The Venn Diagram below illustrates this concept:
The Conditional Entropy H(X∣Y)H(X|Y)H(X∣Y)
 measures, in bits, the Expected Log-Loss associated with variable XXX
 once we have information on variable YYY
:
H(X∣Y)=−∑y∈Yp(y)∑x∈Xp(x∣y)log⁡2(p(x∣y))H(X|Y) = - \sum\limits_{y \in Y} {p(y)\sum\limits_{x \in X} {p(x|y){{\log }_2}} } \left( {p(x|y)} \right)H(X∣Y)=−y∈Y∑​p(y)x∈X∑​p(x∣y)log2​(p(x∣y))
Hence, the Conditional Entropy is a key element in defining the Mutual Information between XXX
 and YYY
.
Note that
I(X,Y)=H(X)−H(X∣Y)I(X,Y) = H(X) - H(X|Y)I(X,Y)=H(X)−H(X∣Y)
is equivalent to:
I(X,Y)=∑x∈X∑y∈Yp(x,y)log⁡2p(x,y)p(x)p(y)I(X,Y) = \sum\limits_{x \in X} {\sum\limits_{y \in Y} {p(x,y){{\log }_2}} } {{p(x,y)} \over {p(x)p(y)}}I(X,Y)=x∈X∑​y∈Y∑​p(x,y)log2​p(x)p(y)p(x,y)​
and furthermore equivalent to:
I(X,Y)=∑y∈Yp(y)∑x∈Xp(x∣y)log⁡2p(x∣y)p(x)I(X,Y) = \sum\limits_{y \in Y} {p(y)\sum\limits_{x \in X} {p(x|y){{\log }_2}} } {{p(x|y)} \over {p(x)}}I(X,Y)=y∈Y∑​p(y)x∈X∑​p(x∣y)log2​p(x)p(x∣y)​
This allows computing the Mutual Information between any two variables.
Usage
For a given network, BayesiaLab can report the Mutual Information in several contexts:
Main Menu > Analysis > Report > Target > Relationship with Target Node.
Note that this table shows the Mutual Information of each node, e.g., XRay, Dyspnea, etc., only with regard to the Target Node, Cancer.
Main Menu > Analysis > Report > Relationship Analysis:
The Mutual Information can also be shown by selecting Main Menu > Analysis > Visual > Overall > Arc > Mutual Information and then clicking the Show Arc Comments icon 
 or selecting Main Menu > View > Show Arc Comments.
Note that the corresponding options under Preferences > Analysis > Visual Analysis > Arc's Mutual Information Analysis have to be selected first:
In Preferences, Child refers to the Relative Mutual Information from the Parent onto the Child node, i.e., in the direction of the arc.
Conversely, Parent refers to the Relative Mutual Information from the Child onto the Parent node, i.e., in the opposite direction of the arc.
Means and Values of Nodes
Normalized Mutual Information
Last modified 1mo ago