Supervised Learning: Markov Blanket
Markov Blanket
Given our objective of predicting the state of the variable , i.e., versus , we define as the Target Node.
We need to specify this explicitly so that the Supervised Learning algorithm can focus on the characterization of the Target Node rather than on a representation of the entire Joint Probability Distribution (JPD) of the learning set.
Upon defining the Target Node, all Supervised Learning algorithms become available under Menus > Learning > Supervised Learning.
Markov Blanket Definition
Upon learning the Markov Blanket for and after having applied the Automatic Layout (shortcut P), the resulting Bayesian network appears as follows:
We can see that the obtained network is a Naive structure on a subset of nodes.
This means that has a direct probabilistic relationship with , , , and . The learned model has four arcs. The figure shows the five connected nodes; unconnected nodes are hidden in the presentation copy.
All other nodes remain unconnected. Their disconnection is a property of this learned model; it is not proof that the corresponding measurements are independent in the population.
Beyond distinguishing between predictors (connected nodes) and non-predictors (disconnected nodes), we can further examine the relationship versus the Target Node by highlighting the Mutual Information of the arcs connecting the nodes.
This function is accessible in Validation Mode F5 by selecting Menus > Analysis > Visual > Overall > Arc > Mutual Information.
Each arc’s thickness is now proportional to the Mutual Information of the nodes it connects. Furthermore, the icon indicates that additional information, i.e., the Arc Comments, is available to be displayed.
So, we select Menus > View > Show Arc Comments. Alternatively, clicking the Show Arc Comment button in the Toolbar achieves the same .
This allows us to examine the Mutual Information between all nodes and the Target Node , which enables us to gauge the relative importance of the nodes.
The current display shows two relative mutual-information percentages for each arc, normalized by the entropy of its endpoint nodes. These percentages are not mutual information in bits or the symmetric normalized measure. The arc thickness provides a visual comparison of the relationships.
Performance Analysis
As we are not equipped with specific domain knowledge about the nodes, we will not further interpret these relationships but rather run an initial test regarding the Network Performance. We want to know how well this Markov Blanket model can predict the states of the diagnosis variable, i.e., Benign versus Malignant. This test is available via Menus > Analysis > Network Performance > Target.
As the analysis starts, BayesiaLab prompts us to specify the Target Evaluation Setting. In the given context, we select Evaluate All States and proceed.
Using the previously defined Test Set for evaluating our model, we obtain the initial performance results, including metrics such as Total Precision, R, R2, etc.
In the context of this example, the table in the center of the report, the so-called Confusion Matrix, is of special interest.
The Confusion Matrix offers Occurrences, Reliability, and Precision tabs. The illustration below makes the Occurrences table explicit: rows are predictions and columns are observed diagnoses.
Of the 40 Malignant test cases, 38 were correctly classified and 2 were predicted as Benign: a false-negative rate of 5%. Of the 73 Benign cases, 71 were correctly classified and 2 were predicted as Malignant.
BayesiaLab reports Overall Precision, or classification accuracy, as the number of correct predictions divided by the test-set size: (71 + 38) / 113 = 96.4602%.
K-Folds Cross-Validation
This result describes one particular held-out test set. A different split can produce different predictions and a different learned structure.
To mitigate any sampling artifacts that may occur in such a one-off Test Set, we can systematically learn networks on a series of different subsets and then aggregate the test results.
For this purpose, we perform a K-Folds Cross-Validation, which will iteratively select K different Learning Sets and Test Sets and then learn the corresponding networks and test their respective performance.
Save a separate working copy for cross-validation, preserving the original trained model and its split. On the cross-validation copy, select Menus > Data > Data Set > Remove Learning/Test Split.
Then, K-Folds Cross-Validation can be started via Menus > Tools > Resampling > Target Evaluation > K-Fold:
We use the same learning algorithm as before, i.e., the Markov Blanket, and choose K=10 as the number of sub-samples to be analyzed.
Use all 569 records, ten folds, Shuffle Samples on, Stratify off, and fixed seed 31. Select Markov Blanket, Delete All Arcs, Variable Selection off, and Evaluate All States; leave Maximum Size of Evidence off. Keep Show Intermediate Results on. The optional output directory preserves each learned fold network.
The settings preview shows 513 learning and 56 test records. Because 569 is not divisible by ten, actual fold sizes differ by one record: a test fold can contain 56 or 57 records, with 513 or 512 remaining for learning. Each record contributes one held-out prediction to the aggregate evaluation.
These runs start from the imported, discretized data. Fold-specific re-discretization was not separately verified, so the reported results should not be interpreted as validation of the complete preprocessing workflow.
The aggregate results are shown below. Overall Precision is 94.7276%: 344 Benign and 195 Malignant records are correctly classified. Seventeen of the 212 Malignant records are predicted as Benign, a false-negative rate of 8.02%. Thirteen Benign records are predicted as Malignant.
Clicking Synthesis Report produces a summary with additional analysis options.
Interpreting the Cross-Validation Results
It is helpful to click the Structure Comparison button to understand what exactly is happening during the K-Folds Cross-Validation.
It brings up a Synthesis Structure of all the networks learned during the K-Folds Cross-Validation.
Black arcs in the Synthesis Structure above indicate that these arcs were present in the Reference Structure (below), i.e., the network that was learned on the basis of the original Learning Set.
The Synthesis Structure summarizes the learned structures relative to the reference. Blue arcs identify relationships absent from the reference. Inspect the native frequency and score information alongside the graph. In this version, the comparison status values can be weight-adjusted; do not read a displayed Count such as 0.9 as an integer number of folds. The following figures illustrate structural variation, without assigning the old edition’s fold counts to this new run.
We can scroll through all the networks discovered during the K-Folds Cross-Validation using the record selector icons . The first structure after the Synthesis Structure is the Reference Structure, which was the current network when we started the K-Folds Cross-Validation.
Comparison Structure 0 is shown below. Compare its connections with the reference; the native status line retains the application’s reported values.
Comparison Structure 1 is shown below. Compare its connections with the reference; the native status line retains the application’s reported values.
Comparison Structure 2 is shown below. Compare its connections with the reference; the native status line retains the application’s reported values.
Comparison Structure 3 is shown below. Compare its connections with the reference; the native status line retains the application’s reported values.
Comparison Structure 4 is shown below. Compare its connections with the reference; the native status line retains the application’s reported values.
The first five comparison structures show that the selected predictors can vary with the training subset. The original model used 456 learning records; the cross-validation training subsets contain 512 or 513. Both the subset composition and its size can affect the learned structure.
Markov Blanket Performance Summary
The Markov Blanket achieved 96.4602% accuracy on the original 113-record test set and 94.7276% in ten-fold cross-validation. These are different evaluations and should not be treated as interchangeable estimates. Next, we apply Augmented Markov Blanket and compare its results under the same evaluation settings.