Bayes' Theorem

Section 4.6 Bayes' Theorem

Conditional probabilities 4.5.3 can be computed using the methods developed above if the appropriate information is available. Some times you will however have some information available, such as

$P(A | B)$ but need

$P(B | A)\text{.}$ The ability to "play around with history" by switching what has been presumed to occur leads to an important result known as Bayes' Theorem.

🔗

Theorem 4.6.1. Bayes' Theorem.

Let $S = \{ S_1, S_2, ... , S_m \}$ where the $S_k$ are pairwise disjoint and $S_1 \cup S_2 \cup ... \cup S_m = S$ (i.e. a partition of the space S). Then for any $A \subset S$

$\begin{equation*} P(S_j | A) = \frac{P(S_j)P(A | S_j)}{\sum_{k=1}^m P(S_k)P(A | S_k)}. \end{equation*}$

The conditional probability $P(S_j | A)$ is called the posterior probability of $S_k\text{.}$

🔗

Proof.

Notice, by the definition of conditional probability 4.5.3 and the multiplication rule 4.5.5

\begin{equation*} P(S_j | A) = \frac{P(S_j \cap A)}{P(A)} = \frac{P(S_j)P( A | S_j)}{P(A)}. \end{equation*}

But using the disjointness of the partition

\begin{align*} P(A) & = P( (A \cap S_1) \cup (A \cup S_2) \cup ... \cup (A \cup S_m) )\\ & = P(A \cap S_1) + P(A \cup S_2) + ... + P(A \cup S_m)\\ & = P(S_1 \cap A) + P(S_2 \cup A) + ... + P(S_m \cup A)\\ & = P(S_1) P(A | S_1) + P(S_2)P(A | S_2) + ... + P(S_m)P(A | S_m)\\ & = \sum_{k=1}^m P(S_k)P(A | S_k) \end{align*}

Put these two expansions together to obtain the desired result.

🔗

To illustrate this result, from the web site http://stattrek.com/probability/bayes-theorem.aspx consider the following problem:

🔗

Marie is getting married tomorrow, at an outdoor ceremony in the desert. In recent years, it has rained only 5 days each year. Unfortunately, the weatherman has predicted rain for tomorrow. When it actually rains, the weatherman correctly forecasts rain 90% of the time. When it doesn't rain, he incorrectly forecasts rain 10% of the time. What is the probability that it will rain on the day of Marie's wedding?

🔗

Notice, all days can be classified into one of two disjoint options:

Rainy, in which case we can deduce from the given info that P(Rain) = 5/365
Not Rainy, and since this is the complement of above, P(Not Rain) = 360/365

🔗

In the notation of Bayes Theorem 4.6.1, let A represent a forecast of Rain and note you have

$\begin{equation*} P(\text{Rain}) = P(S_1) = \frac{5}{365} \end{equation*}$

🔗

and

$\begin{equation*} P(\text{Not Rain}) = P(S_2) = \frac{360}{365}. \end{equation*}$

🔗

Further, you are given the conditional probabilities

$\begin{equation*} P(\text{ Forecast Rain | Rain}) = P( A | S_1) = 0.9 \end{equation*}$

$\begin{equation*} P(\text{ Forecast Rain | Not Rain}) = P( A | S_2) = 0.1 \end{equation*}$

🔗

Notice that the question provided requests that you find the probability of Rain given that the weatherman has forecasted rain. What is given on the other hand is the reverse of that conditional probability. Using Bayes' Theorem allows you to turn this around...

$\begin{align*} P(\text{Rain}) & = P(S_1) P( A | S_1) + P(S_2) P(A | S_2)\\ & = \frac{5}{365} \cdot 0.9 + \frac{360}{365} \cdot 0.1 \end{align*}$

🔗

Hence, putting these together gives

$\begin{align*} P(\text{Rain | Forecast Rain}) & = \frac{\frac{5}{365} \cdot 0.9}{\frac{5}{365} \cdot 0.9 + \frac{360}{365} \cdot 0.1}\\ & = \frac{5 \cdot 0.9}{5 \cdot 0.9 + 360 \cdot 0.1}\\ & = \frac{45}{45+360} \approx 0.111 \end{align*}$

🔗

So, normally there is only approximately a 1.369 percent chance of rain (5/365) on a given day but given that the weatherman has forecast rain, the chance of rain increases to a little more than 11 percent.

🔗

Here's a standard and yet surprising example:

🔗

Your neighbor has 2 children. You learn that he has a daughter Anna. What is the probability that Anna’s sibling is a sister? Your first reaction might be that it is obviously 1/2 since (ok, roughly) it is equally likely that the other sibling was born female or male. But is that accurate?

🔗

If you were to list all of the possible female/male outcomes when having two children, then the sample space is S = {FF, FM, MF, MM} with FM meaning the first child is female and the second is male. Assuming again that girls and boys are equally likely to be born, these 4 outcomes have equal probability of

$1/4\text{.}$

🔗

The question asks is whether the neighbor has another daughter in the set Fem2 = {FF} but since Anna is a girl then the possible outcomes now is only Fem = {FF, FM, MF}.

🔗

So,

$\begin{align*} P(Fem2|Fem) & = \frac{P(Fem2 \cap Fem)}{P(Fem)} \\ & = \frac{P(FF)}{P(FF or FM or MF)}\\ & = \frac{\frac{1}{4}}{\frac{3}{4}} = \frac{1}{3} \end{align*}$

🔗

Bayes' Theorem 4.6.1 also works well in analyzing "Let's Make A Deal" finale choice.

🔗

Checkpoint 4.6.2. WebWork - Bayes'.

You have to be careful to extract the conditional probabilities from the problem.

🔗

Checkpoint 4.6.3. WebWork- Bigger Bayes'.

Notice that having the data expressed in tabular form sometimes makes it easier to deal with.

🔗

The interactive cell below can be used to easily compute all of the conditional probabilities associated with Bayes' Theorem 4.6.1. Notice how the relative size of the pie-shaped partition changes when you presume that an event in the space has already occurred.

xxxxxxxxxx
 
#  This function is used to convert an input string into separate entries
def g(s): 
    S = str(s).replace(',',' ').replace('(',' ').replace(')',' ').split()
    return S
​
@interact
def _(Partition_Probabilities
         =input_box('0.35,0.25,0.40',
                    label="$$ P(S_1),P(S_2),... $$",width=50),
        Conditional_Probabilities
         =input_box('0.02,0.01,0.03',
                    label='$$ P(A|S_1),P(A|S_2),... $$',width=45),
        print_numbers=checkbox(True,label='Numerical Results on Graphs?'),
        auto_update=False):
            
    Partition_Probabilities = g(Partition_Probabilities)
    Conditional_Probabilities = g(Conditional_Probabilities)
    n = len(Partition_Probabilities)
    n0 = len(Conditional_Probabilities)
    
    if (n > n0):
        pretty_print("Unmatched data input.")
        
    else:                       # data streams now are the same size!
        colors = rainbow(n)
        accum = float(0)        #  whether partition probs sum to one
        ends = [0]              # where  graphed partition sectors change
        mid = []                # used for placement of text
        p_Sk_given_A = []       # P( S_k | A )
        pA = 0                  # P(A)
        PP=[]                   # the numerical Partition Probabilities
        CP=[]                   # numerical Conditional Probabilities   
        for k in range(n):
            PP.append(float(Partition_Probabilities[k]))
            CP.append(float(Conditional_Probabilities[k]))
            p_Sk_given_A.append(PP[k]*CP[k] )
            pA += p_Sk_given_A[k]
            accum = accum + PP[k]
            ends.append(accum)
            mid.append((ends[k]+accum)/2)
#
#  From 0 to 1, saving angles for each partition sector boundary.
#  Later, multiple these by 2*pi to get actual sector boundary angles.
#
        if abs(accum-float(1))>0.0000001:     #  Due to roundoff issues
            pretty_print("Sum of probabilities should equal 1.")
        
        else:                           # probability data is sensible
 
#        
#  Venn diagram by drawing sectors from the angles determined above
#  Create a circle of radius 1 to illustrate the the sample space S
#  Sectors with varying colors and print out their names on the edge
#
            G = circle((0,0), 1, rgbcolor='black',fill=False, alpha=0.4,
                       aspect_ratio=True,axes=False,thickness=5)
            for k in range(n):
                G += disk((0,0), 1, (ends[k]*2*pi, ends[k+1]*2*pi),
                          color=colors[mod(k,10)],alpha = 0.2)
                G += text('$S_'+str(k+1)+'$',(1.1*cos(mid[k]*2*pi),
                          1.1*sin(mid[k]*2*pi)), rgbcolor='black')
                
            G += circle((0,0), 0.6, facecolor='yellow', fill = True,
                        alpha = 0.1, thickness=5,edgecolor='black')
    
#  probabilities corresponding to each particular region as a list
            if print_numbers:
​
                html("$P(A) = %s$"%(str(pA),))
                for k in range(n):
                    html("$P(S_{%s} | A)$"%(str(k+1))
                           +"$ = %s$"%str(p_Sk_given_A[k]/pA))
                                        
                    G += text(str(p_Sk_given_A[k]),
                              (0.4*cos(mid[k]*2*pi),
                               0.4*sin(mid[k]*2*pi)), rgbcolor='black')
                    G += text(str(PP[k] - p_Sk_given_A[k]),
                              (0.8*cos(mid[k]*2*pi), 
                               0.8*sin(mid[k]*2*pi)), rgbcolor='black')
        
#  sectors now correspond in area to the Bayes Theorem probabilities
​
            accum = float(0)                        
            ends = [0]       # where the graphed partition sectors change
            mid = []         # middle of each pie chart sector  
            for k in range(n): 
                accum += float(p_Sk_given_A[k]/pA) 
                ends.append(accum)
                mid.append((ends[k]+accum)/2)
            H = circle((0,0), 1, rgbcolor='black',fill=False, 
                        alpha=0,aspect_ratio=True,axes=False,
                       thickness=0)
            H += circle((0,0), 0.6, facecolor='yellow',
                        fill=True,alpha=0.1,
                        aspect_ratio=True,axes=False,
                        thickness=5,edgecolor='black')
            
            for k in range(n):
                H += disk((0,0), 0.6, (ends[k]*2*pi, ends[k+1]*2*pi),
                    color=colors[mod(k,10)],alpha = 0.2)
                H += text('$S_'+str(k+1)+'|A$',
                    (0.7*cos(mid[k]*2*pi), 0.7*sin(mid[k]*2*pi)), 
                    rgbcolor='black')
                    
        #  bayesian probabilities using the smaller set A only
    
            if print_numbers:
                for k in range(n):
                    H += text(str( N(p_Sk_given_A[k]/pA,digits=4) ),
                      (0.4*cos(mid[k]*2*pi), 0.4*sin(mid[k]*2*pi)), 
                      rgbcolor='black')
                    
            G.show(title='Venn diagram of partition with A in middle')
            print
            H.show(title='Venn diagram presuming A has occured')

🔗

You can actually also use Bayes' Theorem 4.6.1 to answer a easy question! Indeed, suppose that you draw one card from a shuffled and standard 52 card deck. Given that you know the card is an Ace, what is the probability that it is also a Heart.

🔗

Using the Bayes' approach, let's break up the world into Hearts (H) and non-Hearts (N). Easily,

$\begin{equation*} P(A|H) = 1/13 \end{equation*}$

$\begin{equation*} P(A|N) = 3/39 \end{equation*}$

🔗

and so by Bayes'

$\begin{equation*} P(H|A) = \frac{P(H) P(A|H)}{P(H) P(A|H) + P(N) P(A|N)} = \frac{\frac{13}{52} \cdot \frac{1}{13}}{\frac{13}{52} \cdot \frac{1}{13} + \frac{39}{52} \cdot \frac{3}{39}} = \frac{1}{4} \end{equation*}$

🔗

as expected!

🔗

Checkpoint 4.6.4. Insured vs Accident.

Your automobile insurance company uses past history to determine how to set rates by measuring the number of accidents caused by clients in various age ranges. The following table summarizes the proportion of those insured and the corresponding probabilities by age range:

Table 4.6.5. Age vs Accident Likelihood

Age	Proportion of Insured	Probability of Accident
16-20	0.05	0.08
21-25	0.06	0.07
26-55	0.49	0.02
55-65	0.25	0.03
over 65	0.15	0.04

One of your family friends insured by this company has an accident.

Determine the conditional probability that the driver was in the 16-20 age range.
Compare this to the probability that the driver was in the 18-20 age range. Discuss the difference.
Determine how much more the company should charge for someone in the 16-20 age range compared to someone in the 26-55 age range.

Solution

Plug the middle column into the first input box and the right column into the second input box of the Bayes Sage Cell.

🔗

Checkpoint 4.6.6. Spinal bifida odds.

Congratulations...your family is having a baby! As part of the prenatal care, some testing is part of the normal procedure including one for spinal bifida (which is a condition in which part of the spinal cord may be exposed.) Indeed, measurement of maternal serum AFP values is a standard tool used in obstetrical care to identify pregnancies that may have an increased risk for this disorder. You want to make plans for the new child's care and want to know how serious to take the test results. However, some times the test indicates that the child has the disorder when in actuality it does not (a false positive) and likewise may indicate that the child does not have the disorder when in fact it does (a false negative.)

The combined accuracy rate for the screen to detect the chromosomal abnormalities mentioned above is approximately 85% with a false positive rate of 5%. This means that (from americanpregnancy.org)

Approximately 85 out of every 100 babies affected by the abnormalities addressed by the screen will be identified. (Positive Positive)
Approximately 5% of all normal pregnancies will receive a positive result or an abnormal level. (False Positive)

Given that your test came back negative, determine the likelihood that the child will actually have spinal bifida.
Given that your test came back negative, determine the likelihood that the child will not have spina bifida
Given that a positive test means you have a 1/100 to 1/300 chance of experiencing one of the abnormalities, determine the likelihood of spinal bifida in a randomly selected child.

You can get some help checking your arithmetic using the Bayes' Sage interact.