Joint Probability & Joint Probability Distribution (JPD)

Definition

A Joint Probability Distribution is the distribution of Joint Probabilities. A Joint Probability, in turn, is the probability of specific values of variables jointly occurring in a domain.

Example

We observe the variables $\mathit{Hair\ Color}$ and $\mathit{Eye\ Color}$ in a population of college students. Here, Joint Probability refers to the probability of specific values for $\mathit{Hair\ Color}$ and $\mathit{Eye\ Color}$ jointly occurring in this population. For instance:

• $P(\mathit{Eye\ Color}=\mathrm{Blue}, \mathit{Hair\ Color}=\mathrm{Blond})=15.86\%$ means that the probability of a student having blue eyes and blond hair in the given population is 15.86%.

• $P(\mathit{Eye\ Color}=\mathrm{Green}, \mathit{Hair\ Color}=\mathrm{Black})=0.85\%$ means that the probability of having green eyes and black hair in that population is only 0.85%.

We can now look across all possible combinations of $\mathit{Hair\ Color}$ and $\mathit{Eye\ Color}$, compute all Joint Probabilities, and list them in a Joint Probability Table, with one row for each combination of the states of the variables. In this example, the size of the Joint Probability Table is manageable:

$\displaystyle \mathit{Number\ of\ States}\, (\mathit{Hair\ Color}) \cdot \mathit{Number\ of\ States}\, (\mathit{Eye\ Color}) = 4 \cdot 4 = 16$

This Joint Probability Table is a direct and complete representation of the Joint Probability Distribution for the variables $\mathit{Hair\ Color}$ and $\mathit{Eye\ Color}$:

Relevance

Since the Joint Probability Distribution covers all possible combinations, it represents all regularities and patterns (or the lack thereof) within a domain. It can be visualized as a high-dimensional hypercube, in which each cell holds the Joint Probability of one complete configuration of the domain.

Because it is complete, any probabilistic query can be answered exactly from the Joint Probability Distribution: marginal probabilities, conditional probabilities, joint probabilities, and the likelihood of evidence are all derivable from it. This is what allows a system built on the Joint Probability Distribution to serve as a true Probabilistic Expert System for Decision Support.

Knowing the Joint Probability Distribution is also required for two key operations in data analysis and inference:

• Marginalization, which is calculating the marginal probability of a variable, e.g., $P(\mathit{Hair\ Color}=\mathrm{Black})=18.25\%$.

• Conditioning, which refers to inferring the values of a variable, given a specific value of another variable, e.g., $P(\mathit{Hair\ Color}=\mathrm{Blond} \mid \mathit{Eye\ Color}=\mathrm{Blue})=43.7\%$.

Challenge

In high-dimensional domains, however, calculating and listing the Joint Probabilities in a Joint Probability Table can become intractable. The size of a Joint Probability Table grows exponentially with the number of variables: if we had 20 variables with 4 states each, the corresponding Joint Probability Table would exceed 1 trillion rows. While the arithmetic is straightforward, the sheer number of calculations can easily exceed the available computational power, both for generating the Joint Probability Table and for performing Marginalization and Conditioning.

“The only way to deal with such large distributions is to constrain the nature of the variable interactions in some manner, both to render specification and ultimately inference in such systems tractable. The key idea is to specify which variables are independent of others, leading to a structured factorisation of the joint probability distribution. Bayesian Belief Networks are a convenient framework for representing such factorisations into local conditional distributions.” (Barber, 2012)

Concretely, a Bayesian network exploits the marginal and conditional independencies in a domain to factor the Joint Probability Distribution into a product of smaller, local conditional probability tables, one per node. This compact representation preserves the reasoning power of the full Joint Probability Distribution while keeping inference computationally tractable. As a result, Bayesian networks are extremely practical for approximating Joint Probability Distributions in complex, high-dimensional problem domains.

References

• Barber, D. (2012). Bayesian Reasoning and Machine Learning. Cambridge: Cambridge University Press. doi:10.1017/CBO9780511804779