51 Confusion Matrix
The Confusion Matrix is simply a Contingency Table with predicted and actual outcomes. Table 51.1 shows the case of the binary (2 by 2) Confusion Matrix for the outcomes that are depicted in Table 50.1. Furthermore, the Confusion Matrix is closely related to the concepts of Sensitivity & Specificity ( Chapter 8) and Bayes Theorem (Chapter 7).
| Actual Fraud: No | Actual Fraud: Yes | |
|---|---|---|
| Predicted Fraud: No | TN = 3 | FN = 2 |
| Predicted Fraud: Yes | FP = 1 | TP = 1 |
From Table 51.1 it can be concluded that:
- the number of True Positives (TP) is equal to 1
- the number of True Negatives (TN) is equal to 3
- the number of False Positives (FP) is equal to 1
- the number of False Negatives (FN) is equal to 2
If the Binomial Classification Model generates perfect predictions then the Confusion Matrix contains zeroes in all off-diagonal cells (i.e. FP = FN = 0). Based on the Confusion Matrix it is possible to define a series of summary statistics that are often used to describe certain aspects of the Binomial Classification Model:
- Total Population
= POP = TP + TN + FP + FN - Prevalence = Actual Positives / Total Population
= (TP + FN) / (TP + TN + FP + FN) - Accuracy = True Predictions / Total Population
= (TP + TN) / (TP + TN + FP + FN) - Precision of Positive Predictions = Correctly Predicted Positives / All Positive Predictions
= TP / (TP + FP) - Precision of Negative Predictions = Correctly Predicted Negatives / All Negative Predictions
= TN / (TN + FN) - False Discovery Rate = False Positive Predictions / All Positive Predictions
= FP / (TP + FP) - False Omission Rate = False Negative Predictions / All Negative Predictions
= FN / (TN + FN) - True Positive Rate = Recall = Sensitivity = Probability of Detection
= TP / (TP + FN) - True Negative Rate = Specificity
= TN / (FP + TN) - False Negative Rate = Miss Rate
= FN / (TP + FN) - False Positive Rate = Fall-out = Probability of False Alarm
= FP / (FP + TN) - Positive Likelihood Ratio = Sensitivity / (1 - Specificity)
= TP / (TP + FN) / (1 - TN / (FP + TN)) - Negative Likelihood Ratio = (1 - Sensitivity) / Specificity
= (1 - TP / (TP + FN)) / (TN / (FP + TN))
Consistent with the discussion of Bayes Theorem in Chapter 7, it is possible to generalise Table 8.1 and include the most important definitions from the items listed above.
| H\(_0\) is true Actual Fraud: No |
H\(_1\) is true Actual Fraud: Yes |
Prevalence = (TP+FN)/(TP+TN+FP+FN) |
|
|---|---|---|---|
| Accept H\(_0\) Predicted Fraud: No |
True Negative (TN) | False Negative (FN) (type II error) |
Precision of Negative Predictions = TN/(TN+FN) |
| Reject H\(_0\) Predicted Fraud: Yes |
False Positive (FP) (type I error) |
True Positive (TP) | Precision of Positive Predictions = TP/(TP+FP) |
| True Negative Rate = TNR = TN/(TN+FP) = Specificity |
True Positive Rate = TPR = TP/(TP+FN) = Sensitivity (Recall) |
Accuracy = (TP+TN)/(TP+TN+FP+FN) |
Besides the ordinary Accuracy measure, several other descriptive statistics are used to summarise the overall quality of the Binomial Classification problem:
- \(F_1\) score which is defined as the Harmonic Mean (Section 57.5) of Sensitivity and Precision of Positive Predictions.
- \(G\) score (G-mean) which is defined as the Geometric Mean (Section 57.4) of Sensitivity and Specificity, i.e. \(\sqrt{\text{Sensitivity} \times \text{Specificity}}\).
- Matthews Correlation Coefficient (Matthews 1975) (or Phi coefficient) (Section 63.7).
- Informedness (also known as Youden’s J statistic (Youden 1950)) measures how well the classifier performs above random chance. It is defined as
= Sensitivity + Specificity − 1
= TP / (TP + FN) + TN / (TN + FP) − 1
A value of 0 indicates the classifier performs no better than random guessing, while a value of 1 indicates perfect classification. - Cohen’s kappa (Cohen 1960) measures agreement between predicted and actual outcomes, adjusted for the agreement that would be expected by chance. It is defined as
= (Accuracy − Expected Accuracy) / (1 − Expected Accuracy)
where Expected Accuracy is the probability that the classifier and actual outcomes would agree by chance alone. A value of 0 indicates agreement no better than chance, while a value of 1 indicates perfect agreement.