Bayes' Theorem
by Maliha Hossain
keyword: probability, Bayes' Theorem, Bayes' Rule
INTRODUCTION
Bayes' Theorem (or Bayes' Rule) allows us to calculate P(A|B) from P(B|A) given that P(A) and P(B) are also known, where A and B are events. In this tutorial, we will derive Bayes' Theorem and illustrate it with a few examples.
Note that this tutorial assumes familiarity with conditional probability and the axioms of probability.
Contents - Bayes' Theorem - Proof - Example 1 - Example 2 - Example 3 - References
Bayes' Theorem
Let $ B_1, B_2, ..., B_n $ be a partition of the sample space $ S $, i.e. $ B_1, B_2, ..., B_n $ are mutually exclusive events whose union equals the sample space S. Suppose that the event $ A $ occurs. Then, by Bayes' Theorem, we have that
$ P[B_j|A] = \frac{P[A|B_j]P[B_j]}{P[A]}, j = 1, 2, . . . , n $
Bayes' Theorem is also often expressed in the following form:
$ P[B_j|A] = \frac{P[A|B_j]P[B_j]}{\sum_{k=1}^n P[A|B_k]P[B_k]} $
Proof
We will now derive Bayes'e Theorem as it is expressed in the second form, which simply takes the expression one step further than the first.
Let $ A $ and $ B_j $ be as defined above. By definition of the conditional probability, we have that
$ P[A|B_j] = \frac{P[A\cap B_j]}{P[B_j]} $
Multiplying both sides with $ B_j $, we get
$ P[A\cap B_j] = P[A|B_j]P[B_j] $
Using the same argument as above, we have that
$ P[B_j|A] = \frac{P[B_j\cap A]}{P[A]} $
$ \Rightarrow P[B_j\cap A] = P[B_j|A]P[A] $
Because of the commutativity property of intersection, we can say that
$ P[B_j|A]P[A] = P[A|B_j]P[B_j] $
Dividing both sides by $ P[A] $, we get
$ P[B_j|A] = \frac{P[A|B_j]P[B_j]}{P[A]} $
Finally, the denominator can be broken down further using the theorem of total probability so that we have the following expression
$ P[B_j|A] = \frac{P[A|B_j]P[B_j]}{\sum_{k=1}^n P[A|B_k]P[B_k]} $
