Hyperparameter Updating
Context
- By their very nature, Bayesian networks in BayesiaLab are nonparametric models.
- However, there is a way to simulate a classical parametric model with a Bayesian network.
- More specifically, you can use nodes to represent the parameters of an equation.
- And the equation, including the parameters, can be encoded as a formula in a node.
- The easiest way to design parametric models consists of using equations for describing the probability distributions of the nodes that are part of the system you are modeling, i.e., what is usually called the parameters, and represent the parameters that are used in the equations, i.e., the hyperparameters, with additional nodes in the network.
Motivation
Example 1: Known Distribution & Parameters
- Suppose we are interested in modeling the distribution of a specific measure obtained in blood tests, let's say, red blood cell count or platelets. The exact nature of this measure is irrelevant in this example.
- We give the variable representing this measure of interest a generic name, i.e., Blood Sample.
- The observed values of Blood Sample range from 0 to 2.5.
- Medical experts believe that the variable Blood Sample follows a normal distribution.
- We also assume the experts know the values of the parameters μ (mean) and σ (standard deviation) of the Normal distribution:
- μ=1
- σ=0.5
- In BayesiaLab, we can define the distribution of Blood Sample with the following probabilistic equation in the Formula panel of the Node Editor:
P(?Blood Sample?) = Normal(?Blood Sample?, 1, 0.5) - The Monitor corresponding to Blood Sample now shows the characteristic shape of a Normal distribution.
Example 2: Known Distribution & Unknown Parameters
-
Suppose the assumption of μ=1 and σ=0.5 no longer holds.
-
Now, the experts only know the ranges of μ and σ but do not know their exact values or distributions.
-
We can represent this uncertainty by creating one node each for μ and σ and consider them hyperparameters.
-
Furthermore, we draw arcs from these hyperparameters nodes to Blood Sample.
-
To encode to complete lack of knowledge regarding the hyperparameters, we specify uniform distributions for μ and σ.
P(?Blood Sample?)=Normal(?Blood Sample?, ?mu?, ?sigma?) -
In Validation Mode
F5, we can see the corresponding distributions:
-
The following networks show various normal distributions after setting arbitrary pieces of evidence on the hyperparameters μ and σ:
-
With such a parametric model, observing the parameter Blood Sample updates the distributions of the hyperparameters via probabilistic inference.
-
However, this approach only produces a set of hyperparameter distributions from a single observation of Blood Sample.
-
Of course, a single observation is insufficient for estimating any hyperparameter reliably.
Bayesian Parameter Updating/Estimation
- To update/estimate the hyperparameter distributions of μ and σ from multiple observations, we must take a different approach.
- We need to perform "Bayesian updating," which is, in fact, the estimation method central to Bayesian statistics.
- Please note that we always emphasize that learning Bayesian networks with BayesiaLab is purely frequentist and that the only thing Bayesian in Bayesian networks is inference.
- In this context, however, we use a Bayesian network to perform Bayesian parameter updating/estimation, i.e., we cross the boundaries between Bayesian networks and Bayesian statistics. In other words, we use a Bayesian network to perform a Bayesian updating/estimation task.
Methodology
- To explain what happens in BayesiaLab's Hyperparameter Updating function, we must break it down into individual steps.
- We can represent hyperparameter updating from multiple observations in the form of a network that consists of multiple observation nodes that are connected to the same set of hyperparameter nodes. We can consider it an "unrolled network."
- The following network is an "unrolled network" and features five observation nodes, i.e., one for each to-be-observed value of Blood Sample.
Example
- Let's assume we have a list of observations of Blood Sample featuring the values, 0.7, 0.6, 0.2, 2.1, and 1.5, which we now add as evidence, one value per observation node:
- As we set values on the observation nodes, we see how the distributions of μ and σ are updated step by step, hence the expression "Bayesian Updating."
- Needless to say, creating a separate node for each new observation would quickly become impractical.
Hyperparameter Updating
- Instead of using an "unrolled network," like the one above, with one node for each observation, BayesiaLab offers a more convenient approach using the Hyperparameter Updating function. It allows a much more compact network representation without any limitations regarding the number of observations.
- To use this function, there are a number of requirements:
- We need to have a network with hyperparameter nodes connected to one observation node, just like we had in Example 2 above.
- We need to assign the hyperparameter nodes to the predefined Class Hyperparameter.
- Select the hyperparameter nodes and then select Contextual Menu > Properties > Classes > Add.
- In the pop-up window, check the radio button Predefined and then choose Hyperparameter from the dropdown menu.
- This tells BayesiaLab that the selected nodes will be updated upon setting evidence on the observation node.
- We also need to add an arc between the hyperparameter nodes μ and σ to represent their conditional dependencies. Note that with any evidence on the observation node, the hyperparameter nodes are independent. However, they become dependent upon setting the first evidence (see Common Child (Collider)).
- The warning triangle on the hyperparameter node for μ indicates that the conditional probability table has not been specified yet.
- For now, we only need to specify a uniform bivariate distribution between the two hyperparameter nodes.
- Go to Contextual Menu > Edit > Probability Distribution and click Normalize.
- Clicking OK confirms this distribution, and the warning disappears from sigma.
- Now we are ready to use this network for Hyperparameter Updating.
- In Validation Mode
F5, selectMain Menu > Inference > Hyperparameter Updating. - The pop-up window now allows you to select the source of evidence:
Hyperparameter Updating- Manual Evidence allows you to set evidence via the Monitors.
- Evidence Scenario File utilizes the Evidence Scenarios associated with your network. This option only shows if Evidence Scenarios are available, as indicated by the icon in the bottom right corner of your window.
- Selecting Data Set utilizes the data associated with your network. This option only shows if a database is available, as indicated by the icon in the bottom right corner of your window.
- We need to have a network with hyperparameter nodes connected to one observation node, just like we had in Example 2 above.
