Bayes theorem S13 - Rhea

Bayes' Theorem

by Maliha Hossain, proud Member of the Math Squad.

keyword: probability, Bayes' Theorem, Bayes' Rule 

INTRODUCTION

Bayes' Theorem (or Bayes' Rule) allows us to calculate P(A|B) from P(B|A) given that P(A) and P(B) are also known. In this tutorial, we will derive Bayes' Theorem and illustrate it with a few examples. After going over the examples, if you have any questions or if you find any mistakes please leave me a comment at the end of the relevant section.

Note that this tutorial assumes familiarity with conditional probability and the axioms of probability. If you interested in the derivation of the conditional distributions for continuous and discrete random variables, you may wish to go over Professor Mary Comer's notes on the subject.

 Contents
- Bayes' Theorem
- Proof
- Example Problems
- References

Bayes' Theorem

Let $ B_1, B_2, ..., B_n $ be a partition of the sample space $ S $, i.e. $ B_1, B_2, ..., B_n $ are mutually exclusive events whose union equals the sample space S. Suppose that the event $ A $ occurs. Then, by Bayes' Theorem, we have that

$ P[B_j|A] = \frac{P[A|B_j]P[B_j]}{P[A]}, j = 1, 2, . . . , n $

Bayes' Theorem is also often expressed in the following form:

$ P[B_j|A] = \frac{P[A|B_j]P[B_j]}{\sum_{k=1}^n P[A|B_k]P[B_k]} $

Proof

We will now derive Bayes' Theorem as it is expressed in the second form, which simply takes the expression one step further than the first.

Let $ A $ and $ B_j $ be as defined above. By definition of the conditional probability, we have that

$ P[A|B_j] = \frac{P[A\cap B_j]}{P[B_j]} $

Multiplying both sides with $ B_j $, we get

$ P[A\cap B_j] = P[A|B_j]P[B_j] \ $

Using the same argument as above, we have that

$ \begin{align} P[B_j|A] & = \frac{P[B_j\cap A]}{P[A]} \\ \Rightarrow P[B_j\cap A] &= P[B_j|A]P[A] \end{align} $

Because of the commutativity property of intersection, we can say that

$ P[B_j|A]P[A] = P[A|B_j]P[B_j] \ $

Dividing both sides by $ P[A] $, we get

$ P[B_j|A] = \frac{P[A|B_j]P[B_j]}{P[A]} $

Finally, the denominator can be broken down further using the theorem of total probability so that we have the following expression

$ P[B_j|A] = \frac{P[A|B_j]P[B_j]}{\sum_{k=1}^n P[A|B_k]P[B_k]} $

Example Problems

Example 1: Quality Control

Example 2: False Positive Paradox

Example 3: Monty Hall Problem

References

• Alberto Leon-Garcia, Probability, Statistics, and Random Processes for Electrical Engineering, Third Edition

Questions and comments

If you have any questions, comments, etc. please post them below:

• Comment / question 1

Back to Math Squad page