bayesia logo
BayesiaLab
Analysis Network Performance Overall Learning Test Set

Network Performance Analysis Overall — Learning & Test Set

Context

  • This Overall Performance Report evaluates a network with regard to a dataset that does have a Learning/Test Set split.
  • If your dataset does not have a Learning/Test Set split, please see Report for Learning Set.
  • Given that most performance measures here are the same as in the Report for Learning Set, we refer to that topic when appropriate rather than duplicating the content.
  • In this topic, we focus on the additional features and objectives related to the Learning/Test Set split.

Notation

  • BB denotes the Bayesian network to be evaluated.
  • DD represents the entire Dataset associated with the Bayesian network BB. The Dataset DD is split into two partitions:
    • DLD_L represents the Learning Set of the Dataset from which network BB was learned.
    • DTD_T represents the Test Set (or holdout sample), i.e., the portion of the Dataset that will be used for evaluation network BB.
    • ELE_L represents an n-dimensional observation (Evidence), i.e., one row or record in the Learning Set DLD_L, from which the Bayesian network BB was learned.
    • ETE_T represents an n-dimensional observation (Evidence), i.e., one row or record in the Test Set DTD_T, which will be used to evaluate network BB.
    • NLN_L refers to the number of observations ELE_L in the Learning Set DLD_L.
    • NTN_T refers to the number of observations ETE_T in the Test Set DTD_T.
  • CC refers to a Complete or fully connected network, in which all nodes have a direct link to all other nodes. Therefore, the complete network CC is an exact representation of the chain rule. As such, it does not utilize any conditional independence assumptions for representing the Joint Probability Distribution.
  • UU represents an Unconnected network, in which there are no connections between nodes, which means that all nodes are marginally independent.

Example

To explain and illustrate the Overall Performance Report, we use a Bayesian network model that was generated with one of BayesiaLab's Unsupervised Learning algorithms. This network is available for download here:

ElPasoLT.xbl

Overall Performance Report

The Report window consists of three tabs

  • Test Dataset
  • Learning Dataset
  • Comparison

which feature two views each:

  • Density Function
    • The x-axis represents the Log-Loss values in increasing order.
    • The y-axis shows the probability density for each Log-Loss value on the x-axis.
  • Distribution Function
    • The observations EE in the dataset DD are sorted in ascending order according to their Log-Loss values:
      • The x-axis shows the observation number.
      • The y-axis shows the Log-Loss value corresponding to each observation.
Test Dataset EvaluationLearning Dataset EvaluationComparison

The radio buttons on the bottom-left of the window allow you to switch the view between the Density function (Histogram) and the Distribution function.

Either view provides a visualization of the Log-Loss values for all observations in the dataset DD given the to-be-evaluated Bayesian network BB. Thus, the plots provide you with a visual representation of how well the network BB fits the dataset DD.

Comparing Density & Distribution Functions

In the topic, Overall Performance Report for the Learning Set, the focus was primarily on how well the network BB fits dataset DD. All measures were about goodness-of-fit.

Our objective for this topic is broader. We are still looking for a good fit, but also want to understand how well the learned network model generalizes beyond the dataset from which it was learned.

In this context, the Comparison tab provides a key visual. You can see the histograms of the Log-Losses of the Learning Set and Test Set overlaid on top of each other.

Log-Loss Computation

The computation of Log-Loss values is at the very core of this Overall Performance Report. In the Report for Learning Set, the Log-Loss values were computed for the entire dataset DD.

Given the Learning/Test Set split, BayesiaLab now needs to compute all metrics separately for the Learning Set and the Test Set.

And, in addition to Log-Loss LLB(E)LL_B(E) for the to-be-evaluated network BB, BayesiaLab also needs to compute the Log-Loss values LLC(E)LL_C(E) for the complete network CC and Log-Loss LLU(E)LL_U(E) and the unconnected network UU.

So, to produce the plots and all related metrics, BayesiaLab has to perform the following computations:

  • LLB(EL)LL_B(E_L), the Log-Loss value for each observation/evidence in the Learning Set based on the learned and to be-evaluated Bayesian network B.
  • LLC(EL)LL_C(E_L), the Log-Loss value for each observation/evidence in the Learning Set based on the complete network CC.
  • LLU(EL)LL_U(E_L), the Log-Loss value for each observation/evidence in the Learning Set based on the unconnected network UU.
  • LLB(ET)LL_B(E_T), the Log-Loss value for each observation/evidence in the Test Set based on the learned and to be-evaluated Bayesian network BB.
  • LLC(ET)LL_C(E_T), the Log-Loss value for each observation/evidence in the Test Set based on the complete network CC.
  • LLU(ET)LL_U(E_T), the Log-Loss value for each observation/evidence in the Test Set based on the unconnected network UU.

The following Log-Loss Table is an extract of the first ten rows each from the Learning Set DLD_L and the Test Set DTD_T along with the computed Log-Loss values for each record:

Log-Loss Table

Learning/TestMonthHourTemperatureShortwave Radiation (W/m²)Wind Speed (m/s)Energy Demand (MWh)Log-Loss (Bayesian Network)Log-Loss (Complete Network)Log-Loss (Unconnected Network)
LLB(EL)L{L_B}(E_L)LLC(EL)L{L_C}(E_L)LLU(EL)LL_U(E_L)
learning81836.57213.602.001574.0013.4314.6821.96
learning81936.04105.911.901574.0013.8514.6821.61
learning82034.7142.722.141485.0012.1311.8719.41
learning82133.940.002.751470.0011.8811.8717.71
learning82233.190.003.551378.0011.8911.0917.72
learning82332.380.004.211249.0014.1212.6816.93
learning8031.560.004.501110.0013.0512.3616.94
learning8229.660.004.90975.0011.2211.6814.66
learning8329.020.004.60944.0010.9111.3614.66
learning8527.160.003.11927.0011.2910.9814.65
Entropy HB(DL)H_B(D_L)Entropy HC(DL)H_C(D_L)Entropy HU(DL)H_U(D_L)
Mean13.1612.49
Std. Dev.2.061.34
Minimum9.699.43
Maximum30.8814.68
Normalized68.369%64.893%
Learning/TestMonthHourTemperatureShortwave Radiation (W/m²)Wind Speed (m/s)Energy Demand (MWh)Log-Loss (Bayesian Network)Log-Loss (Complete Network)Log-Loss (Unconnected Network)
LLB(ET)LL_B(E_T)LLC(ET)LL_C(E_T)LLU(ET)LL_U(E_T)
test8130.600.004.801031.0013.4114.6816.90
test8428.160.003.70926.0010.7410.6814.68
test81533.20318.622.561554.0014.6212.6820.63
test81832.71192.242.131468.0013.7414.6819.37
test8027.090.004.751113.0011.0911.0916.44
test8125.870.007.531033.0013.6912.6817.62
test8423.270.008.90928.0015.8214.6815.77
test8523.100.006.82928.0011.7111.5115.04
test81428.61353.333.191412.0019.10?19.35
test82027.7027.594.121459.0012.3912.0919.00
Entropy HB(DT)H_B(D_T)Entropy HC(DT)H_C(D_T)Entropy HU(DT)H_U(D_T)
Mean13.2312.41
Std. Dev.2.121.31
Minimum9.699.43
Maximum26.0714.68
Normalized68.725%64.499%

The complete Log-Loss Table serves as the basis for calculating the measures that are reported at the bottom of the report.

The measures reported at the bottom of the Test Set and Learning Set tab, are shown side by side on the Comparison tab:

For clarity, we match up the report's labels to the notation introduced at the beginning of this topic and the corresponding definitions.

Label in ReportDefinition in Context of Test SetDefinition in Context of Learning Set
Entropy (H), i.e., mean of Log-Loss Values of all observationsHB(DT)=LLB(ET){H_B}({D_T}) = \overline{L{L_B}({E_T})}HB(DL)=LLB(EL){H_B}({D_L}) = \overline{L{L_B}({E_L})}
Normalized Entropy (Hn)HBN(DT)=HB(DT)log2(SD){H_{BN}}({D_T}) = \frac{{{H_B}({D_T})}}{{{{\log }_2}({S_D})}}HBN(DL)=HB(DL)log2(SD){H_{BN}}({D_L}) = \frac{{{H_B}({D_L})}}{{{{\log }_2}({S_D})}}
Hn(Complete)HCN(DT)=HC(DT)log2(SD){H_{CN}}({D_T}) = \frac{{{H_C}({D_T})}}{{{{\log }_2}({S_D})}}HCN(DL)=HC(DL)log2(SD){H_{CN}}({D_L}) = \frac{{{H_C}({D_L})}}{{{{\log }_2}({S_D})}}
Hn(Unconnected)HUN(DT)=HU(DT)log2(SD){H_{UN}}({D_T}) = \frac{{{H_U}({D_T})}}{{{{\log }_2}({S_D})}}HUN(DL)=HU(DL)log2(SD){H_{UN}}({D_L}) = \frac{{{H_U}({D_L})}}{{{{\log }_2}({S_D})}}
Contingency Table FitCTFB=100×HU(DT)HB(DT)HU(DT)HC(DT){CTF_B} = 100 \times \frac{{{H_U}({D_T}) - {H_B}({D_T})}}{{{H_U}({D_T}) - {H_C}({D_T})}}CTFB=100×HU(DL)HB(DL)HU(DL)HC(DL){CTF_B} = 100 \times \frac{{{H_U}({D_L}) - {H_B}({D_L})}}{{{H_U}({D_L}) - {H_C}({D_L})}}
DevianceDevB(DT)=2N×ln(2)×(HB(DT)HC(DT))De{v_B}({D_T}) = 2N \times \ln(2) \times \left( {H_B}({D_T}) - {H_C}({D_T}) \right)DevB(DL)=2N×ln(2)×(HB(DL)HC(DL))De{v_B}({D_L}) = 2N \times \ln(2) \times \left( {H_B}({D_L}) - {H_C}({D_L}) \right)
Number of Processed ObservationsN(DT)N(D_T)N(DL)N(D_L)

Impossible Observations

Impossible Observations refer to observations/evidence ETE_T in the Test Set DTD_T, which are entirely incompatible with the network BB that was learned from the Learning Set DLD_L.

Kolmogorov-Smirnov Test

The Kolmogorov-Smirnov Test (KS Test) is a common tool for comparing statistical distributions. Here, we use it to compare the distribution similarity between two samples, i.e. learning and test, so this test is only computed for the comparison panel.

More specifically, it compares the distributions of the Log-Losses. The values Z, D and the corresponding p-value are displayed.

Extract Data Set

The final element in the report window is the Extract Data Set button. This is a practical tool for identifying and examining outliers, e.g., those at the far end of the right tail of the histogram.

  • Clicking the Extract Data Set button brings up a new window that allows you to extract observations from the dataset according to the criteria you define:
  • Right Tail Extraction selects the specified percentage of observations, beginning with the highest Log-Loss value.
  • Interval Extraction allows you to specify a lower and upper boundary of Log-Loss values to be included.
  • Upon selecting either method and clicking OK, you are prompted to choose a file name and location.
  • BayesiaLab saves the observations that meet the criteria in csv format.
  • Note that the Log-Loss values that are used for extraction are not included in the saved dataset.

Copyright © 2024 Bayesia S.A.S., Bayesia USA, LLC, and Bayesia Singapore Pte. Ltd. All Rights Reserved.