Joint Probability & Joint Probability Distribution (JPD)

Definition

• A Joint Probability Distribution is the distribution of Joint Probabilities.

• A Joint Probability is the probability of specific values of variables jointly occurring in a domain.

Example

• We observe the variables Hair Color and Eye Color in a population of college students.

• Joint Probability refers to the probability of specific values for Hair Color and Eye Color jointly occurring in this population.

• For instance,

  • P(Eye Color=Blue, Hair Color=Blond)=15.86% means that the probability of a student having blue eyes and blond hair in the given population is 15.86%.

  • P(Eye Color=Green, Hair Color=Black)=0.85% means that the probability of having green eyes and black hair in that population is only 0.85%.

• We can now look across all possible combinations of Hair Color and Eye Color, compute all Joint Probabilities and list them in a Joint Probability Table, with one row for each combination of the states of the variables.

• In this example, the size of the Joint Probability Table is manageable: Number of States (Hair Color) × Number of States (Eye Color) = 4 × 4 = 16

• This Joint Probability Table is a direct and complete representation of the Joint Probability Distribution for the variables Hair Color and Eye Color:

  Hair Color	Eye Color	Joint Probability
  Black	Brown	11.49%
  Brown	Brown	20.10%
  Red	Brown	4.39%
  Blond	Brown	1.18%
  Black	Blue	3.38%
  Brown	Blue	14.19%
  Red	Blue	2.87%
  Blond	Blue	15.88%
  Black	Hazel	2.53%
  Brown	Hazel	9.12%
  Red	Hazel	2.36%
  Blond	Hazel	1.69%
  Black	Green	0.84%
  Brown	Green	4.90%
  Red	Green	2.36%
  Blond	Green	2.70%
  	Sum	100.00%

Relevance

• As the Joint Probability Distribution covers all possible combinations, it represents all regularities and patterns (or the lack thereof) within a domain.

• Knowing the Joint Probability Distribution is required for performing two key operations for data analysis and inference:

  • Marginalization, which is calculating the marginal probability of a variable, e.g., P(Hair Color=Black)=18.25%.

  • Conditioning, which refers to inferring the values of a variable, given a specific value of another variable, e.g., P(Hair Color=Blond | Eye Color=Blue)=43.7%.

Challenge

• In high-dimensional domains, however, calculating and listing the Joint Probabilities in a Joint Probability Table can become intractable.

• The size of a Joint Probability Table grows exponentially with the number of variables. For example, if we had 20 variables with 4 states each, the size of the corresponding Joint Probability Table would exceed 1 trillion rows.

• While the arithmetic is straightforward, the sheer number of calculations can easily exceed the available computational power, both for generating the Joint Probability Table as well as for performing Marginalization and Conditioning.

• "The only way to deal with such large distributions is to constrain the nature of the variable interactions in some manner, both to render specification and ultimately inference in such systems tractable. The key idea is to specify which variables are independent of others, leading to a structured factorisation of the joint probability distribution. [Bayesian] Belief Networks are a convenient framework for representing such factorisations into local conditional distributions." (Barber, 2012)

• This means that Bayesian networks are extremely practical for approximating Joint Probability Distributions in complex, high-dimensional problem domains.

References

• Barber, D. (2012). Bayesian Reasoning and Machine Learning. Cambridge: Cambridge University Press. doi:10.1017/CBO9780511804779