59  Confusion Matrix

The Confusion Matrix is simply a Contingency Table with predicted and actual outcomes. Table 59.1 shows the case of the binary (2 by 2) Confusion Matrix for the outcomes that are depicted in Table 58.1. Furthermore, the Confusion Matrix is closely related to the concepts of Sensitivity & Specificity ( Chapter 8) and Bayes Theorem (Chapter 7).

Table 59.1: Confusion Matrix -- Predicted by Actual Fraud

	Actual Fraud: No	Actual Fraud: Yes
Predicted Fraud: No	TN = 3	FN = 2
Predicted Fraud: Yes	FP = 1	TP = 1

From Table 59.1 it can be concluded that:

• the number of True Positives (TP) is equal to 1
• the number of True Negatives (TN) is equal to 3
• the number of False Positives (FP) is equal to 1
• the number of False Negatives (FN) is equal to 2

If the Binomial Classification Model generates perfect predictions then the Confusion Matrix contains zeroes in all off-diagonal cells (i.e. FP = FN = 0). Based on the Confusion Matrix it is possible to define a series of summary statistics that are often used to describe certain aspects of the Binomial Classification Model:

• Total Population
  = POP = TP + TN + FP + FN
• Prevalence = Actual Positives / Total Population
  = (TP + FN) / (TP + TN + FP + FN)
• Accuracy = True Predictions / Total Population
  = (TP + TN) / (TP + TN + FP + FN)
• Precision of Positive Predictions = Correctly Predicted Positives / All Positive Predictions
  = TP / (TP + FP)
• Precision of Negative Predictions = Correctly Predicted Negatives / All Negative Predictions
  = TN / (TN + FN)
• False Discovery Rate = False Positive Predictions / All Positive Predictions
  = FP / (TP + FP)
• False Omission Rate = False Negative Predictions / All Negative Predictions
  = FN / (TN + FN)
• True Positive Rate = Recall = Sensitivity = Probability of Detection
  = TP / (TP + FN)
• True Negative Rate = Specificity
  = TN / (FP + TN)
• False Negative Rate = Miss Rate
  = FN / (TP + FN)
• False Positive Rate = Fall-out = Probability of False Alarm
  = FP / (FP + TN)
• Positive Likelihood Ratio = Sensitivity / (1 - Specificity)
  = TP / (TP + FN) / (1 - TN / (FP + TN))
• Negative Likelihood Ratio = (1 - Sensitivity) / Specificity
  = (1 - TP / (TP + FN)) / (TN / (FP + TN))

Consistent with the discussion of Bayes Theorem in Chapter 7, it is possible to generalise Table 8.1 and include the most important definitions from the items listed above.

Table 59.2: Binomial Classification

	H\(_0\) is true Actual Fraud: No	H\(_1\) is true Actual Fraud: Yes	Prevalence = (TP+FN)/(TP+TN+FP+FN)
Accept H\(_0\) Predicted Fraud: No	True Negative (TN)	False Negative (FN) (type II error)	Precision of Negative Predictions = TN/(TN+FN)
Reject H\(_0\) Predicted Fraud: Yes	False Positive (FP) (type I error)	True Positive (TP)	Precision of Positive Predictions = TP/(TP+FP)
	True Negative Rate = TNR = TN/(TN+FP) = Specificity	True Positive Rate = TPR = TP/(TP+FN) = Sensitivity (Recall)	Accuracy = (TP+TN)/(TP+TN+FP+FN)

Besides the ordinary Accuracy measure, several other descriptive statistics are used to summarise the overall quality of the Binomial Classification problem:

• \(F_1\) score which is defined as the Harmonic Mean (Section 65.5) of Sensitivity and Precision of Positive Predictions.
• \(G\) score (G-mean) which is defined as the Geometric Mean (Section 65.4) of Sensitivity and Specificity, i.e. \(\sqrt{\text{Sensitivity} \times \text{Specificity}}\).
• Matthews Correlation Coefficient (Matthews 1975) (or Phi coefficient) (Section 71.7).
• Informedness (also known as Youden’s J statistic (Youden 1950)) measures how well the classifier performs above random chance. It is defined as
  = Sensitivity + Specificity − 1
  = TP / (TP + FN) + TN / (TN + FP) − 1
  A value of 0 indicates the classifier performs no better than random guessing, while a value of 1 indicates perfect classification.
• Cohen’s kappa (Cohen 1960) measures agreement between predicted and actual outcomes, adjusted for the agreement that would be expected by chance. It is defined as
  = (Accuracy − Expected Accuracy) / (1 − Expected Accuracy)
  where Expected Accuracy is the probability that the classifier and actual outcomes would agree by chance alone. A value of 0 indicates agreement no better than chance, while a value of 1 indicates perfect agreement.

Cohen, Jacob. 1960. “A Coefficient of Agreement for Nominal Scales.” Educational and Psychological Measurement 20 (1): 37–46. https://doi.org/10.1177/001316446002000104.

Matthews, B. W. 1975. “Comparison of the Predicted and Observed Secondary Structure of T4 Phage Lysozyme.” Biochimica Et Biophysica Acta (BBA) – Protein Structure 405 (2): 442–51. https://doi.org/10.1016/0005-2795(75)90109-9.

Youden, W. J. 1950. “Index for Rating Diagnostic Tests.” Cancer 3 (1): 32–35. https://doi.org/10.1002/1097-0142(1950)3:1<32::AID-CNCR2820030106>3.0.CO;2-3.