33 Bayes’ rule
Summary
Bayes’ rule is a method for calculating the conditional probability of an event.
Bayes’ rule gives the probability of outcome A given B, P(A|B), using the unconditional probability of outcome A, the probability of observing B given A and the total probability of outcome B:
P(A|B)=\frac{P(B|A)P(A)}{P(B)}
- The total probability P(B) can be calculated as:
P(B) = P(B|A)P(A) + P(B|\neg A)P(\neg A)
- Bayes’ rule can be thought of as a way to update beliefs about a hypothesis H given evidence E:
P(H|E) = \frac{P(E|H)P(H)}{P(E)}
33.1 Introduction
Bayes’ rule is a method for estimating the conditional probability of an event.
Specifically, Bayes’ rule allows us to use the following information to estimate the conditional probability of outcome A given outcome B:
- The unconditional probability of outcome A
- The probability of observing outcome B given outcome A
- The total probability of outcome B.
The formula for Bayes’ rule is:
\begin{align*} P(A|B)&=\frac{P(A\cap B)}{P(B)} \\[12pt] &=\frac{P(B|A)P(A)}{P(B)} \end{align*}The denominator P(B) is the total probability of event B. If the total probability of event B is not directly available, we can often calculate it with information concerning the conditional probabilities of B given the occurrence (or not) of A.
P(B)=P(B|A)P(A)+P(B|\neg A)P(\neg A)
The symbol \neg represents “not”.
We can therefore write Bayes’ rule as follows:
\begin{align*} P(A|B)&=\frac{P(B|A)P(A)}{P(B)} \\[12pt] &=\frac{P(B|A)P(A)}{P(B|A)P(A)+P(B|\neg A)P(\neg A)} \end{align*}33.2 Updating beliefs
We can think of Bayes’ rule as how we should update our beliefs in light of a new event.
Rational agents should update their beliefs using Bayes’ rule.
In this case, the following elements are involved:
A hypothesis, H. For example, “the coin is fair” or “the coin is rigged”.
The prior probability of the hypothesis H being true, P(H). For example, “the coin is fair” has a prior probability of 0.5.
The probability of observing event E given a hypothesis H, P(E|H). For example, “the coin shows a head” has a probability of 0.5 given that the coin is fair.
The posterior probability of the belief H given the event E, P(H|E). For example, we would have an updated probability in our hypothesis that “the coin is fair” based on the coin showing a head.
Under this framing, Bayes’ rule is formulated as follows:
\begin{align*} \underbrace{P(H|E)}_\text{Posterior belief}&=\frac{P(E|H)\overbrace{P(H)}^\text{Prior belief}}{P(E)} \\[12pt] &=\frac{P(E|H)P(H)}{P(E|H)P(H)+P(E|\neg H)P(\neg H)} \end{align*}33.3 A rigged coin
Suppose your friend has two coins. One is a fair coin with a head on one side and a tail on the other. The second coin is a rigged coin with a head on both sides.
Your friend takes one of the coins and flips it. The coin shows a head. What is the probability that this coin is the rigged coin?
We will assume that he randomly selected either coin with a probability of 50%. We take that as our prior belief:
P(\text{rigged})=0.5
The probability of a head if it is the rigged coin is 1.
P(\text{head}|\text{rigged})=1
To use Bayes’ rule, we need the total probability that a head comes up, P(\text{head}).
Here we will use the formula for total probability.
\begin{align*} P(\text{head})&=P(\text{head}|\text{rigged})P(\text{rigged})+P(\text{head}|\text{fair})P(\text{fair}) \\[6pt] &=1\times 0.5+0.5\times 0.5 \\[6pt] &=0.75 \end{align*}Putting this into Bayes’ rule:
\begin{align*} P(\text{rigged}|\text{head})&=\frac{P(\text{head}|\text{rigged})P(\text{rigged})}{P(\text{head})} \\[12pt] &=\frac{1\times 0.5}{0.75} \\[6pt] &=\frac{2}{3} \end{align*}Your friend flips the coin again and gets another head. What is the updated probability that the coin is rigged?
The prior belief is now P(\text{rigged})=\frac{2}{3}.
The total probability of flipping a head is:
\begin{align*} P(\text{head})&=P(\text{head}|\text{rigged})P(\text{rigged})+P(\text{head}|\text{fair})P(\text{fair}) \\[6pt] &=1\times \frac{2}{3}+0.5\times \frac{1}{3} \\[6pt] &=\frac{5}{6} \end{align*}Putting this into Bayes rule:
\begin{align*} P(\text{rigged}|\text{head})&=\frac{P(\text{head}|\text{rigged})P(\text{rigged})}{P(\text{head})} \\[12pt] &=\frac{1\times \frac{2}{3}}{\frac{5}{6}} \\[6pt] &=\frac{4}{5} \end{align*}Your belief that the coin is rigged has now increased to 80%.
Your friend flips the coin 10 more times and gets 10 more heads. What is the updated probability that the coin is rigged?
We use our prior belief of P(\text{rigged})=\frac{4}{5}.
The total probability of flipping 10 heads is:
\begin{align*} P(\text{10 heads})&=P(\text{10 heads}|\text{rigged})P(\text{rigged})+P(\text{10 heads}|\text{fair})P(\text{fair}) \\[6pt] &=1\times \frac{4}{5}+\bigg(\frac{1}{2}\bigg)^{10}\times \frac{1}{5} \\[6pt] &=0.8002 \end{align*}Putting this into Bayes’ rule:
\begin{align*} P(\text{rigged}|\text{10 heads})&=\frac{P(\text{10 heads}|\text{rigged})P(\text{rigged})}{P(\text{10 heads})} \\[12pt] &=\frac{1\times \frac{4}{5}}{0.8002} \\[12pt] &=0.99976 \end{align*}We now believe the coin is rigged with greater than 99.9% probability.
33.4 Balls from an urn
You have two urns filled with balls. Urn 1 has 30% black balls and 70% yellow balls. Urn 2 has 70% black balls and 30% yellow balls. The labels have fallen off the urns, so you do not know which urn is which.
You reach into one of the urns and pull out a yellow ball. What is the probability that you have drawn the ball from urn 1?
The Bayes’ rule formula to solve this problem is:
\begin{align*} P(\text{urn 1}|\text{yellow})=\frac{P(\text{yellow}|\text{urn 1})P(\text{urn 1})}{P(\text{yellow})} \end{align*}We take the prior probability of the ball coming from urn 1 to be 50%. The probability of drawing a yellow ball from urn 1 is 70%.
The total probability of drawing a yellow ball is:
\begin{align*} P(\text{yellow})&=P(\text{yellow}|\text{urn 1})P(\text{urn 1})+P(\text{yellow}|\text{urn 2})P(\text{urn 2}) \\[6pt] &=0.3\times 0.5+0.7\times 0.5 \\[6pt] &=0.5 \end{align*}Putting this into Bayes’ rule:
\begin{align*} P(\text{urn 1}|\text{yellow})&=\frac{P(\text{yellow}|\text{urn 1})P(\text{urn 1})}{P(\text{yellow})} \\[12pt] &=\frac{P(\text{yellow}|\text{urn 1})P(\text{urn 1})}{P(\text{yellow}|\text{urn 1})P(\text{urn 1})+P(\text{yellow}|\text{urn 2})P(\text{urn 2})} \\[12pt] &=\frac{0.7\times 0.5}{0.7\times 0.5+0.3\times 0.5} \\[12pt] &=0.7 \end{align*}You put the first ball back in the urn, reach in again and pull out a black ball. What is the probability that you have drawn the ball from urn 1?
Given we have already drawn one ball and updated our probability, we will use the prior probability of P(\text{urn 1})=0.7.
The total probability of drawing a black ball is:
\begin{align*} P(\text{black})&=P(\text{black}|\text{urn 1})P(\text{urn 1})+P(\text{black}|\text{urn 2})P(\text{urn 2}) \\[6pt] &=0.3\times 0.7+0.7\times 0.3 \\[6pt] &=0.42 \end{align*}Putting this into Bayes rule:
\begin{align*} P(\text{urn 1}|\text{black})&=\frac{P(\text{black}|\text{urn 1})P(\text{urn 1})}{P(\text{black})} \\[12pt] &=\frac{0.3\times 0.7}{0.42} \\[12pt] &=0.5 \end{align*}The answer of 0.5 should seem intuitive. We have now drawn one black and one yellow ball. In combination, this is uninformative and we are back at our initial prior of 0.5.
33.5 The Monty Hall problem
Recall the Monty Hall problem:
Suppose you’re on a game show and you’re given the choice of three doors: Behind one door is a car; behind the others, goats. You pick a door, say No. 1, and the host, who knows what’s behind the doors, opens another door, say No. 3, which has a goat. He then says to you, “Do you want to pick door No. 2?” Is it to your advantage to switch your choice?
Assume that the rules of this game show are that:
The host must always open a door that you did not choose.
The host must always open a door to reveal a goat and never the car.
The host must always offer you the choice to switch between the chosen door and the remaining closed door.
We want to know the probability that the car is behind Door 2 given the host opened Door 3. We want to know P(C2|D3). 1
To determine this using Bayes’ rule, we would use the following formula:
\begin{align*} P(C2|D3)&=\frac{P(D3|C2)P(C2)}{P(D3)} \end{align*}P(D3) is the probability that the host opens Door 3. It is calculated using the formula for total probability:
\begin{align*} P(D3)&=P(D3|C1)P(C1)+P(D3|C2)P(C2)+P(D3|C3)P(C3) \end{align*}Each of those elements are as follows.
P(C1), P(C2) and P(C3) are our prior probability of the car being behind each door, which is \frac{1}{3}.
P(D3|C1) is the probability that the host opens door 3, given the car is behind door 1. The host could open either of Door 2 or Door 3 as neither has the car behind it, so the probability of Door 3 is \frac{1}{2}.
P(D3|C2) is the probability that the host opens Door 3, given the car is behind Door 2. The host must open that door, so the probability is one. They cannot open the door you have chosen or the door that the car is behind.
P(D3|C3) is the probability that the host opens door 3, given the car is behind door 3. The host cannot open a door to show the car, so the probability is zero.
Returning to our equations, the total probability of the host opening Door 3 is:
\begin{align*} P(D3)&=P(D3|C1)P(C1)+P(D3|C2)P(C2)+P(D3|C3)P(C3) \\[6pt] &=\frac{1}{2}\times\frac{1}{3}+1\times\frac{1}{3}+0\times\frac{1}{3} \\[6pt] &=\frac{1}{2} \end{align*}Now we can calculate the probability that the car is behind Door 2, given the host opened Door 3:
\begin{align*} P(C2|D3)&=\frac{P(D3|C2)P(C2)}{P(D3)} \\[12pt] &=\frac{1\times\frac{1}{3}}{\frac{1}{2}} \\[12pt] &=\frac{2}{3} \end{align*}The contestant should switch doors.
Technically, we want P(C2|D3\cap X1) where X1 is our selection of Door 1. However, adding this complication to the calculation does not change the answer.↩︎