# Mutual Information

The Mutual Information

$I(X, Y)$

measures the amount of information gained on variable $X$

(the reduction in the Expected Log-Loss) by observing variable $Y$

:

$I(X,Y) = H(X) - H(X|Y)$

The

**Venn Diagram**below illustrates this concept:The Conditional Entropy

$H(X|Y)$

measures, in bits, the Expected Log-Loss associated with variable $X$

once we have information on variable $Y$

:$H(X|Y) = - \sum\limits_{y \in Y} {p(y)\sum\limits_{x \in X} {p(x|y){{\log }_2}} } \left( {p(x|y)} \right)$

Hence, the Conditional Entropy is a key element in defining the Mutual Information between

$X$

and $Y$

.Note that

$I(X,Y) = H(X) - H(X|Y)$

is equivalent to:

$I(X,Y) = \sum\limits_{x \in X} {\sum\limits_{y \in Y} {p(x,y){{\log }_2}} } {{p(x,y)} \over {p(x)p(y)}}$

and furthermore equivalent to:

$I(X,Y) = \sum\limits_{y \in Y} {p(y)\sum\limits_{x \in X} {p(x|y){{\log }_2}} } {{p(x|y)} \over {p(x)}}$

This allows computing the Mutual Information between any two variables.

For a given network, BayesiaLab can report the Mutual Information in several contexts:

`Main Menu > Analysis > Report > Target > Relationship with Target Node`

.- Note that this table shows the Mutual Information of each node, e.g., XRay, Dyspnea, etc., only with regard to the Target Node, Cancer.

`Main Menu > Analysis > Report > Relationship Analysis`

:

- The Mutual Information can also be shown by selecting
`Main Menu > Analysis > Visual > Overall > Arc > Mutual Information`

and then clicking the Show Arc Comments icon or selecting`Main Menu > View > Show Arc Comments`

.

- Note that the corresponding options under
`Preferences > Analysis > Visual Analysis > Arc's Mutual Information Analysis`

have to be selected first:

- In Preferences, Child refers to the Relative Mutual Information from the Parent onto the Child node, i.e., in the direction of the arc.
- Conversely, Parent refers to the Relative Mutual Information from the Child onto the Parent node, i.e., in the opposite direction of the arc.

Last modified 6mo ago