7 Bayes’ Theorem
7.1 Theorem
As shown in Equation 5.1
\[ \begin{equation} \text{P}(A | B) = \frac{\text{P}(A B)}{\text{P}(B)} \end{equation} \tag{7.1}\]
and
\[ \begin{equation} \text{P}(B | A) = \frac{\text{P}(B A)}{\text{P}(A)} \end{equation} \tag{7.2}\]
Since \(\text{P}(A B)\) = \(\text{P}(B A)\) it follows by substituting Equation 7.1 into Equation 7.2 that
\[ \begin{equation} \text{P}(A | B) = \frac{\text{P}(B | A) \text{P}(A)}{\text{P}(B)} \end{equation} \tag{7.3}\]
Equation 7.3 is the simplest formulation of Bayes’ Theorem which shows that the probability of \(A\) given \(B\) can be expressed as a function of the probability of \(B\) given \(A\). The LHS of Equation 7.3 is called the posterior probability, \(\text{P}(B | A)\) is called the likelihood, \(\text{P}(A)\) is the prior probability, and \(\text{P}(B)\) is the marginal likelihood (evidence).
It is obvious that Equation 7.3 can be rewritten as \(\text{P}(H | D) \propto \text{P}(D | H) \text{P}(H)\) where \(A\) has been replaced by \(H\) (Hypothesis) and \(B\) by \(D\) (observed Data).
In other words: the probability of the hypothesis being true, given the fact that specific data have been observed, varies with the likelihood of the observed data when the hypothesis is true times the prior probability (subjective, prior probability) of the hypothesis.
By extending Equation 7.3, another useful equation can be found. In fact one only has to think of two different hypotheses \(H_1\) and \(H_2\) and apply Equation 7.3:
\[ \begin{equation} \frac{\text{P}(H_1 | D)}{\text{P}(H_2 | D)} = \frac{\text{P}(D | H_1)}{\text{P}(D | H_2)} \frac{\text{P}(H_1)}{\text{P}(H_2)} \end{equation} \tag{7.4}\]
or in words: the posterior odds are equal to the likelihood ratio multiplied by the prior odds (Bayes’ theorem).
7.2 Example
Suppose that there are two sacks of gold and silver coins. Both have been shuffled thoroughly. Denote the first sack as \(H_1\) and the second as \(H_2\). Furthermore, it is known that in the first sack there are 150 gold coins and only 50 silver coins, in the other sack there are 100 and 200 coins respectively.
Suppose that a (blindfolded) person had the opportunity to draw a coin out of one sack, and suppose that the drawn coin would be a gold coin. What is the probability of the drawn coin to come from the first sack?
Assuming that both sacks have the same chance of being picked (the person is blindfolded) this problem can be solved quite easily by applying Bayes’ theorem as follows:
\(\frac{\text{P}(H_1 | D = Gold)}{\text{P}(H_2 | D = Gold)} = \frac{\frac{150}{200}}{\frac{100}{300}} \frac{\frac{1}{2}}{\frac{1}{2}} = \frac{9}{4}\)
Hence, \(\text{P}(H_1 | D = Gold) = \frac{9}{13}\) because \(\text{P}(H_1 | D = Gold) + \text{P}(H_2 | D = Gold) = 1\).
An important issue, when applying Bayes’ theorem in practice is that there always is a (subjective) prior probability that is being incorporated in solving the problem (in our example we assumed that a priori both sacks had an equal chance of being picked).
For a full decision-oriented Bayesian workflow (posterior thresholds, Bayes factors, and decision error interpretation), see Chapter 105.