BayesiaLab
Analysis Network Performance Overall Learning Test Set

Network Performance Analysis Overall — Learning & Test Set

Context

• This Overall Performance Report evaluates a network with regard to a dataset that does have a Learning/Test Set split.
• If your dataset does not have a Learning/Test Set split, please see Report for Learning Set.
• Given that most performance measures here are the same as in the Report for Learning Set, we refer to that topic when appropriate rather than duplicating the content.
• In this topic, we focus on the additional features and objectives related to the Learning/Test Set split.

Notation

• $B$ denotes the Bayesian network to be evaluated.
• $D$ represents the entire Dataset associated with the Bayesian network $B$. The Dataset $D$ is split into two partitions:
• $D_L$ represents the Learning Set of the Dataset from which network $B$ was learned.
• $D_T$ represents the Test Set (or holdout sample), i.e., the portion of the Dataset that will be used for evaluation network $B$.
• $E_L$ represents an n-dimensional observation (Evidence), i.e., one row or record in the Learning Set $D_L$, from which the Bayesian network $B$ was learned.
• $E_T$ represents an n-dimensional observation (Evidence), i.e., one row or record in the Test Set $D_T$, which will be used to evaluate network $B$.
• $N_L$ refers to the number of observations $E_L$ in the Learning Set $D_L$.
• $N_T$ refers to the number of observations $E_T$ in the Test Set $D_T$.
• $C$ refers to a Complete or fully connected network, in which all nodes have a direct link to all other nodes. Therefore, the complete network $C$ is an exact representation of the chain rule. As such, it does not utilize any conditional independence assumptions for representing the Joint Probability Distribution.
• $U$ represents an Unconnected network, in which there are no connections between nodes, which means that all nodes are marginally independent.

Example

To explain and illustrate the Overall Performance Report, we use a Bayesian network model that was generated with one of BayesiaLab's Unsupervised Learning algorithms. This network is available for download here:

ElPasoLT.xbl

Overall Performance Report

The Report window consists of three tabs

• Test Dataset
• Learning Dataset
• Comparison

which feature two views each:

• Density Function
• The x-axis represents the Log-Loss values in increasing order.
• The y-axis shows the probability density for each Log-Loss value on the x-axis.
• Distribution Function
• The observations $E$ in the dataset $D$ are sorted in ascending order according to their Log-Loss values:
• The x-axis shows the observation number.
• The y-axis shows the Log-Loss value corresponding to each observation.
Test Dataset EvaluationLearning Dataset EvaluationComparison

The radio buttons on the bottom-left of the window allow you to switch the view between the Density function (Histogram) and the Distribution function.

Either view provides a visualization of the Log-Loss values for all observations in the dataset $D$ given the to-be-evaluated Bayesian network $B$. Thus, the plots provide you with a visual representation of how well the network $B$ fits the dataset $D$.

Comparing Density & Distribution Functions

In the topic, Overall Performance Report for the Learning Set, the focus was primarily on how well the network $B$ fits dataset $D$. All measures were about goodness-of-fit.

Our objective for this topic is broader. We are still looking for a good fit, but also want to understand how well the learned network model generalizes beyond the dataset from which it was learned.

In this context, the Comparison tab provides a key visual. You can see the histograms of the Log-Losses of the Learning Set and Test Set overlaid on top of each other.

Log-Loss Computation

The computation of Log-Loss values is at the very core of this Overall Performance Report. In the Report for Learning Set, the Log-Loss values were computed for the entire dataset $D$.

Given the Learning/Test Set split, BayesiaLab now needs to compute all metrics separately for the Learning Set and the Test Set.

And, in addition to Log-Loss $LL_B(E)$ for the to-be-evaluated network $B$, BayesiaLab also needs to compute the Log-Loss values $LL_C(E)$ for the complete network $C$ and Log-Loss $LL_U(E)$ and the unconnected network $U$.

So, to produce the plots and all related metrics, BayesiaLab has to perform the following computations:

• $LL_B(E_L)$, the Log-Loss value for each observation/evidence in the Learning Set based on the learned and to be-evaluated Bayesian network B.
• $LL_C(E_L)$, the Log-Loss value for each observation/evidence in the Learning Set based on the complete network $C$.
• $LL_U(E_L)$, the Log-Loss value for each observation/evidence in the Learning Set based on the unconnected network $U$.
• $LL_B(E_T)$, the Log-Loss value for each observation/evidence in the Test Set based on the learned and to be-evaluated Bayesian network $B$.
• $LL_C(E_T)$, the Log-Loss value for each observation/evidence in the Test Set based on the complete network $C$.
• $LL_U(E_T)$, the Log-Loss value for each observation/evidence in the Test Set based on the unconnected network $U$.

The following Log-Loss Table is an extract of the first ten rows each from the Learning Set $D_L$ and the Test Set $D_T$ along with the computed Log-Loss values for each record:

Log-Loss Table

Learning/TestMonthHourTemperatureShortwave Radiation (W/m²)Wind Speed (m/s)Energy Demand (MWh)Log-Loss (Bayesian Network)Log-Loss (Complete Network)Log-Loss (Unconnected Network)
$L{L_B}(E_L)$$L{L_C}(E_L)$$LL_U(E_L)$
learning81836.57213.602.001574.0013.4314.6821.96
learning81936.04105.911.901574.0013.8514.6821.61
learning82034.7142.722.141485.0012.1311.8719.41
learning82133.940.002.751470.0011.8811.8717.71
learning82233.190.003.551378.0011.8911.0917.72
learning82332.380.004.211249.0014.1212.6816.93
learning8031.560.004.501110.0013.0512.3616.94
learning8229.660.004.90975.0011.2211.6814.66
learning8329.020.004.60944.0010.9111.3614.66
learning8527.160.003.11927.0011.2910.9814.65
Entropy $H_B(D_L)$Entropy $H_C(D_L)$Entropy $H_U(D_L)$
Mean13.1612.49
Std. Dev.2.061.34
Minimum9.699.43
Maximum30.8814.68
Normalized68.369%64.893%
Learning/TestMonthHourTemperatureShortwave Radiation (W/m²)Wind Speed (m/s)Energy Demand (MWh)Log-Loss (Bayesian Network)Log-Loss (Complete Network)Log-Loss (Unconnected Network)
$LL_B(E_T)$$LL_C(E_T)$$LL_U(E_T)$
test8130.600.004.801031.0013.4114.6816.90
test8428.160.003.70926.0010.7410.6814.68
test81533.20318.622.561554.0014.6212.6820.63
test81832.71192.242.131468.0013.7414.6819.37
test8027.090.004.751113.0011.0911.0916.44
test8125.870.007.531033.0013.6912.6817.62
test8423.270.008.90928.0015.8214.6815.77
test8523.100.006.82928.0011.7111.5115.04
test81428.61353.333.191412.0019.10?19.35
test82027.7027.594.121459.0012.3912.0919.00
Entropy $H_B(D_T)$Entropy $H_C(D_T)$Entropy $H_U(D_T)$
Mean13.2312.41
Std. Dev.2.121.31
Minimum9.699.43
Maximum26.0714.68
Normalized68.725%64.499%

The complete Log-Loss Table serves as the basis for calculating the measures that are reported at the bottom of the report.

The measures reported at the bottom of the Test Set and Learning Set tab, are shown side by side on the Comparison tab:

For clarity, we match up the report's labels to the notation introduced at the beginning of this topic and the corresponding definitions.

Label in ReportDefinition in Context of Test SetDefinition in Context of Learning Set
Entropy (H), i.e., mean of Log-Loss Values of all observations${H_B}({D_T}) = \overline{L{L_B}({E_T})}$${H_B}({D_L}) = \overline{L{L_B}({E_L})}$
Normalized Entropy (Hn)${H_{BN}}({D_T}) = \frac{{{H_B}({D_T})}}{{{{\log }_2}({S_D})}}$${H_{BN}}({D_L}) = \frac{{{H_B}({D_L})}}{{{{\log }_2}({S_D})}}$
Hn(Complete)${H_{CN}}({D_T}) = \frac{{{H_C}({D_T})}}{{{{\log }_2}({S_D})}}$${H_{CN}}({D_L}) = \frac{{{H_C}({D_L})}}{{{{\log }_2}({S_D})}}$
Hn(Unconnected)${H_{UN}}({D_T}) = \frac{{{H_U}({D_T})}}{{{{\log }_2}({S_D})}}$${H_{UN}}({D_L}) = \frac{{{H_U}({D_L})}}{{{{\log }_2}({S_D})}}$
Contingency Table Fit${CTF_B} = 100 \times \frac{{{H_U}({D_T}) - {H_B}({D_T})}}{{{H_U}({D_T}) - {H_C}({D_T})}}$${CTF_B} = 100 \times \frac{{{H_U}({D_L}) - {H_B}({D_L})}}{{{H_U}({D_L}) - {H_C}({D_L})}}$
Deviance$De{v_B}({D_T}) = 2N \times \ln(2) \times \left( {H_B}({D_T}) - {H_C}({D_T}) \right)$$De{v_B}({D_L}) = 2N \times \ln(2) \times \left( {H_B}({D_L}) - {H_C}({D_L}) \right)$
Number of Processed Observations$N(D_T)$$N(D_L)$

Impossible Observations

Impossible Observations refer to observations/evidence $E_T$ in the Test Set $D_T$, which are entirely incompatible with the network $B$ that was learned from the Learning Set $D_L$.

Kolmogorov-Smirnov Test

The Kolmogorov-Smirnov Test (KS Test) is a common tool for comparing statistical distributions. Here, we use it to compare the distribution similarity between two samples, i.e. learning and test, so this test is only computed for the comparison panel.

More specifically, it compares the distributions of the Log-Losses. The values Z, D and the corresponding p-value are displayed.

Extract Data Set

The final element in the report window is the Extract Data Set button. This is a practical tool for identifying and examining outliers, e.g., those at the far end of the right tail of the histogram.

• Clicking the Extract Data Set button brings up a new window that allows you to extract observations from the dataset according to the criteria you define:
• Right Tail Extraction selects the specified percentage of observations, beginning with the highest Log-Loss value.
• Interval Extraction allows you to specify a lower and upper boundary of Log-Loss values to be included.
• Upon selecting either method and clicking OK, you are prompted to choose a file name and location.
• BayesiaLab saves the observations that meet the criteria in csv format.
• Note that the Log-Loss values that are used for extraction are not included in the saved dataset.

For North America

Bayesia USA

4235 Hillsboro Pike
Suite 300-688
Nashville, TN 37215, USA

+1 888-386-8383
info@bayesia.us

Bayesia S.A.S.

Parc Ceres, Batiment N 21
rue Ferdinand Buisson
53810 Change, France

For Asia/Pacific

Bayesia Singapore

1 Fusionopolis Place
#03-20 Galaxis
Singapore 138522