Comparing Mutual Information and Arc Force

Mutual Information

The Mutual Information between two variables XX and YY is defined as follows:

I(X,Y)=โˆ‘xโˆˆXโˆ‘yโˆˆYp(x,y)logโก2p(x,y)p(x)p(y)I(X,Y)=\sum_{x \in X}\sum_{y \in Y} p(x,y)\log_2 \frac{p(x,y)}{p(x)p(y)}

Arc Force

The Kullback-Leibler Divergence (or KL Divergence) is used to measure the strength of the relationship between two nodes that are directly connected by an arc.

We commonly refer to the KL Divergence as Arc Force.

Formally, โ€‹the Kullback-Leibler Divergence DKL{D_{KL}} measures the difference between two distributions PP and QQ.

DKL(P(X)โˆฅQ(X))=โˆ‘XP(X)log2P(X)Q(X)D_{KL}(P({\cal X})\|Q({\cal X}))=\sum_{\cal X}P({\cal X})log_2\frac{P({\cal X})}{Q({\cal X})}

For our purposes, we consider PP the Bayesian network that does include the arc for which we wish to compute the Arc Force, and QQ the Bayesian network that does not contain that arc but is otherwise identical.

We interpret this difference DKL{D_{KL}} as the "force of the arc" or Arc Force.

Comparing Mutual Information and Arc Force

Mutual Information can be rewritten as:

I(X,Y)=DKL(p(x,y)โˆฅp(x)p(y))I(X,Y)=D_{KL}(p(x,y)\|p(x)p(y))

Therefore, Mutual Information II and Arc Force DKL{D_{KL}} are identical if there are no spouses (co-parents) involved in the relationship of interest.

Example 1

Let's consider the following network consisting of two nodes, X and Z.

The Conditional Probability Table associated with the node Z is defined as follows:

We now analyze this relationship in terms of Mutual Information in Validation Mode using Main Menu > Analysis > Visual > Overall > Arc > Arcs' Mutual Information and click on the Arc Comments icon in the Toolbar.

The top number in the box shows the Mutual Information II.

The bottom number in the box is Symmetric Normalized Mutual Information ISN{I_{SN}}.

ISN(X,Y)=2ร—I(X,Y)logโก2(SX)+logโก2(SY){I_{SN}}(X,Y) = 2 \times \frac{{I(X,Y)}}{{{{\log }_2}({S_X}) + {{\log }_2}({S_Y})}}

Next, we now analyze this relationship in terms of Arc Force using Main Menu > Analysis > Visual > Overall > Arc > Kullback-Leibler and, again, click on the Arc Comments icon in the Toolbar.

The top number in the box shows the Arc Force DKL{D_{KL}}.

The bottom number, in blue, represents the relative weight of this arc compared to the sum of all Arc Forces in the network. Given that this network consists only of one arc, this arc's weight accounts for 100%.

So, for now, both analyses return the same value, i.e., 0.3436. As we stated above, Mutual InformationI and Arc Force DKL{D_{KL}}are identical with regard to an arc if no spouses (co-parents) are involved in the relationship of interest.

Example 2

However, as soon as we have spouses (co-parents) involved, the Arc Force provides a more comprehensive characterization of the relationship.

Let's consider the following deterministic example, in which node Z represents an Exclusive-OR (XOR) gate with regard to its inputs X and Y.

The Truth Table associated with the node Z is defined as follows:

We now analyze this relationship in terms of Mutual Information in Validation Mode using Main Menu > Analysis > Visual > Overall > Arc > Arcs' Mutual Information and click on the Arc Comments icon in the Toolbar.

We can easily validate this assessment by simulating evidence for X and Y individually.

Indeed, there is no impact of either X and Y on Z.

Comparison with Arc Force (Kullback-Leibler Divergence)

Next, we now analyze this relationship in terms of Arc Force using Main Menu > Analysis > Visual > Overall > Arc > Kullback-Leibler and, again, click on the Arc Comments icon in the Toolbar.

The Arc Force, which takes into account the network as a whole, reveals the perfectly-deterministic relationship between X, Y, and Z.

Last updated

Logo

Bayesia USA

info@bayesia.us

Bayesia S.A.S.

info@bayesia.com

Bayesia Singapore

info@bayesia.com.sg

Copyright ยฉ 2024 Bayesia S.A.S., Bayesia USA, LLC, and Bayesia Singapore Pte. Ltd. All Rights Reserved.