Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
The Symmetric Normalized Mutual Information measure takes the difference of the respective entropies of X and Y into account:
For a given network, BayesiaLab can report the Symmetric Normalized Mutual Information in several contexts:
Main Menu > Analysis > Report > Relationship Analysis
:
Note that the corresponding options under Preferences > Analysis > Visual Analysis > Arc's Mutual Information Analysis
have to be selected first:
In Preferences, Child refers to the Relative Mutual Information from the Parent onto the Child node, i.e., in the direction of the arc.
Conversely, Parent refers to the Relative Mutual Information from the Child onto the Parent node, i.e., in the opposite direction of the arc.
The following Venn Diagram illustrates that the Mutual Information is symmetrical for the two variables and , i.e., .
However, the variables and can each have a different number of states. Therefore, their respective entropies can be very different.
This means that the absolute value of Mutual Information cannot be interpreted without context. In the Venn Diagram, for instance, reduces by a bigger percentage than does . As such, would be more "important" with regard to than it would be with regard to .
As a result, we have an easy-to-interpret measure that relates to both and together.
The Symmetric Normalized Mutual Information can also be shown by selecting Main Menu > Analysis > Visual > Overall > Arc > Mutual Information
and then clicking the Show Arc Comments icon or selecting Main Menu > View > Show Arc Comments
.
Symmetric Relative Mutual Information computes the percentage of information gained by observing and :
This normalization is calculated similarly to Pearson's Correlation Coefficient .
where denotes variance.
So, Mutual Information is comparable to covariance, and Entropy is analogous to variance.
For a given network, BayesiaLab can report the Symmetric Relative Mutual Information in several contexts:
Main Menu > Analysis > Report > Relationship Analysis
:
Note that the corresponding options under Preferences > Analysis > Visual Analysis > Arc's Mutual Information Analysis
have to be selected first:
In Preferences, Child refers to the Relative Mutual Information from the Parent onto the Child node, i.e., in the direction of the arc.
Conversely, Parent refers to the Relative Mutual Information from the Child onto the Parent node, i.e., in the opposite direction of the arc.
The Mutual Information measures the amount of information gained on variable (the reduction in the Expected Log-Loss) by observing variable :
The Venn Diagram below illustrates this concept:
The Conditional Entropy measures, in bits, the Expected Log-Loss associated with variable once we have information on variable :
Hence, the Conditional Entropy is a key element in defining the Mutual Information between and .
Note that
is equivalent to:
and furthermore equivalent to:
This allows computing the Mutual Information between any two variables.
For a given network, BayesiaLab can report the Mutual Information in several contexts:
Main Menu > Analysis > Report > Target > Relationship with Target Node
.
Note that this table shows the Mutual Information of each node, e.g., XRay, Dyspnea, etc., only with regard to the Target Node, Cancer.
Main Menu > Analysis > Report > Relationship Analysis
:
Note that the corresponding options under Preferences > Analysis > Visual Analysis > Arc's Mutual Information Analysis
have to be selected first:
In Preferences, Child refers to the Relative Mutual Information from the Parent onto the Child node, i.e., in the direction of the arc.
Conversely, Parent refers to the Relative Mutual Information from the Child onto the Parent node, i.e., in the opposite direction of the arc.
The Symmetric Normalized Mutual Information can also be shown by selecting Main Menu > Analysis > Visual > Overall > Arc > Mutual Information
and then clicking the Show Arc Comments icon or selecting Main Menu > View > Show Arc Comments
.
The Mutual Information can also be shown by selecting Main Menu > Analysis > Visual > Overall > Arc > Mutual Information
and then clicking the Show Arc Comments icon or selecting Main Menu > View > Show Arc Comments
.
To provide some intuition for the Arc Force and Node Force measures computed by BayesiaLab, we use the water hose and balloon metaphor:
Imagine that we have a Bayesian network in which the variables are balloons and the arcs are elastic, perforated water hoses. The size of the holes in the hose represents the uncertainty contained in the conditional probability table associated with the child node.
For a deterministic relationship (i.e., we know the state of one variable given the state of the other one with certainty), there are no holes at all in the hose, and therefore, no water is lost between these two nodes.
Conversely, for an entirely uncertain relationship, in which information on one variable does not yield any information regarding the other one (such a “relationship” cannot be machine-learned as there is no correlation in the dataset), the size of the holes would be so large that no water could be transmitted from one node the other.
Now, we are sending a constant flow of water into this system. The thickness of a hose represents the actual water flow and is inversely proportional to the size of its holes. Big holes mean that most water leaks, and the effective water flow is minimal.
The pressure in a balloon, and therefore its size, depends on the number of connected hoses and the sizes of their respective holes.
BayesiaLab's Mapping function visualizes Node Force and Arc Force so you can easily identify the most important variables in a network, even in high-dimensional spaces.
In the networks below, for instance, the most important nodes are Country, Age, and Gender:
Based on Mutual Information, Normalized Mutual Information includes a normalization factor:
where denotes the number of states of .
This means that the Mutual Information is divided by the maximum possible entropy of , i.e., .
With that, the formal definition of Normalized Mutual Information is:
BayesiaLab reports the Normalized Mutual Information in the Target Analysis Report: Main Menu > Analysis > Report > Target > Relationship with Target Node
.
Note that this table shows the Normalized Mutual Information of each node, e.g., XRay, Dyspnea, etc., with regard to the Target Node, Cancer.
In Preferences, Child refers to the Normalized Mutual Information from the Parent onto the Child node, i.e., in the direction of the arc.
Conversely, Parent refers to the Normalized Mutual Information from the Child onto the Parent node, i.e., in the opposite direction of the arc.
The Normalized Mutual Information can also be shown by selecting Main Menu > Analysis > Visual > Overall > Arc > Mutual Information
and then clicking the Show Arc Comments icon or selecting Main Menu > View > Show Arc Comments
.
Note that the corresponding options under Main Menu > Preferences > Analysis > Visual Analysis > Arc's Mutual Information Analysis
have to be selected first:
Based on Mutual Information, Relative Mutual Information is defined as:
Relative Mutual Information expresses in percent how much the entropy (or uncertainty) of is reduced by observing .
In older versions of BayesiaLab, Relative Mutual Information was also called Normalized Mutual Information.
Please see the up-to-date definition of Normalized Mutual Information.
BayesiaLab reports the Relative Mutual Information in the Target Analysis Report: Main Menu > Analysis > Report > Target > Relationship with Target Node
.
Note that this table shows the Relative Mutual Information of each node, e.g., XRay, Dyspnea, etc., only with regard to the Target Node, Cancer.
Note that the corresponding options under Preferences > Analysis > Visual Analysis > Arc's Mutual Information Analysis
have to be selected first:
In Preferences, Child refers to the Relative Mutual Information from the Parent onto the Child node, i.e., in the direction of the arc.
Conversely, Parent refers to the Relative Mutual Information from the Child onto the Parent node, i.e., in the opposite direction of the arc.
The Kullback-Leibler Divergence (or KL Divergence) is used to measure the strength of the relationship between two nodes that are directly connected by an arc.
We commonly refer to the KL Divergence as Arc Force.
Mutual Information can be rewritten as:
Let's consider the following network consisting of two nodes, X and Z.
The Conditional Probability Table associated with the node Z is defined as follows:
The bottom number, in blue, represents the relative weight of this arc compared to the sum of all Arc Forces in the network. Given that this network consists only of one arc, this arc's weight accounts for 100%.
However, as soon as we have spouses (co-parents) involved, the Arc Force provides a more comprehensive characterization of the relationship.
Let's consider the following deterministic example, in which node Z represents an Exclusive-OR (XOR) gate with regard to its inputs X and Y.
The Truth Table associated with the node Z is defined as follows:
We can easily validate this assessment by simulating evidence for X and Y individually.
Indeed, there is no impact of either X and Y on Z.
The Arc Force, which takes into account the network as a whole, reveals the perfectly-deterministic relationship between X, Y, and Z.
The Relative Mutual Information can also be shown by selecting Main Menu > Analysis > Visual > Overall > Arc > Mutual Information
and then clicking the Show Arc Comments icon or selecting Main Menu > View > Show Arc Comments
.
The between two variables and is defined as follows:
Formally, the Kullback-Leibler Divergence measures the difference between two distributions and .
For our purposes, we consider the Bayesian network that does include the arc for which we wish to compute the Arc Force, and the Bayesian network that does not contain that arc but is otherwise identical.
We interpret this difference as the "force of the arc" or Arc Force.
Therefore, Mutual Information and Arc Force are identical if there are no spouses (co-parents) involved in the relationship of interest.
We now analyze this relationship in terms of Mutual Information in Validation Mode using Main Menu > Analysis > Visual > Overall > Arc > Arcs' Mutual Information
and click on the Arc Comments icon in the Toolbar.
The top number in the box shows the Mutual Information .
The bottom number in the box is Symmetric Normalized Mutual Information .
Next, we now analyze this relationship in terms of Arc Force using Main Menu > Analysis > Visual > Overall > Arc > Kullback-Leibler
and, again, click on the Arc Comments icon in the Toolbar.
The top number in the box shows the Arc Force .
So, for now, both analyses return the same value, i.e., 0.3436. As we stated above, Mutual InformationI and Arc Force are identical with regard to an arc if no spouses (co-parents) are involved in the relationship of interest.
We now analyze this relationship in terms of Mutual Information in Validation Mode using Main Menu > Analysis > Visual > Overall > Arc > Arcs' Mutual Information
and click on the Arc Comments icon in the Toolbar.
Next, we now analyze this relationship in terms of Arc Force using Main Menu > Analysis > Visual > Overall > Arc > Kullback-Leibler
and, again, click on the Arc Comments icon in the Toolbar.