Information Gain

Definition
The Information Gain regarding evidence EEE
 is the difference between the:
Log-Loss LLU(E)L{L_U}\left( E \right)LLU​(E)
, given an unconnected network UUU
, i.e., a so-called straw model, in which all nodes are marginally independent;
Log-Loss LLB(E)L{L_B}\left( E \right)LLB​(E)
 given a reference network BBB
.
​IGB(E)=log⁡2(P(e1,...,en)∏i=1nP(ei))=LLU(E)−LLB(E)IG_B(E) = {\log _2}\left( {{{P({e_1},...,{e_n})} \over {\prod\limits_{i = 1}^n {P({e_i})} }}} \right) = L{L_U}(E) - L{L_B}(E)IGB​(E)=log2​⎝⎛​i=1∏n​P(ei​)P(e1​,...,en​)​⎠⎞​=LLU​(E)−LLB​(E)
​
In earlier versions of BayesiaLab, Information Gain was named Consistency.
Interpretation
The Log-Loss reflects the "cost" in bits of applying the network BBB
 to evidence EEE
, i.e., the number of bits that are needed to encode evidence EEE
. The lower the probability of evidence EEE
, the higher the Log-Loss.
As a result, a positive value of Information Gain would reflect a "cost-saving" for encoding evidence EEE
 by virtue of having network BBB
. In other words, encoding EEE
 with network BBB
 is less "costly" than encoding it with the straw model UUU
. Therefore, evidence EEE
 would be consistent with network BBB
.
Conversely, a negative Information Gain indicates a so-called conflict, Log-Loss of evidence EEE
 is higher with the straw model UUU
 compared to the reference network BBB
. Note that conflicting evidence does not necessarily mean that the reference network is wrong. Rather, it probably indicates that such a set of evidence belongs to the tail of the distribution that is represented by the reference network BBB
.
However, if evidence EEE
 is drawn from the original data on which the reference network BBB
 was originally learned, the probability of observing conflicting evidence should be smaller than the probability of observing consistent evidence.
So, for a network model to be useful, there should generally be more sets of evidence with a positive Information Gain, i.e., consistent observations, than sets of evidence with a negative Information Gain, i.e., conflicting observations. 
Therefore, the mean value of the Information Gain of a reference network BBB
 compared to a straw model UUU
 is a useful performance indicator of the reference network BBB
. 
Related BayesiaLab Functions
Information Gain and Evidence Analysis
Network Performance
Kullback-Leibler Divergence (Arc Force)
Contingency Table Fit
Last modified 1mo ago