Confusion Matrix

A confusion matrix is an $N\times N$ matrix, where $N$ is the number of predicted classes. For a binary prediction the confusion matrix is a 2 by 2 matrix.

It is not a score on its own. It is a table of counts, and every other predictive metric is calculated from those counts.

Binary Confusion Matrix

A 2 by 2 matrix features 4 different combinations of predicted and actual values.

Outcome	Meaning
True Positive (TP)	accurately predicted positive values
True Negative (TN)	accurately predicted negative values
False Positive (FP) (Type 1 Error)	negative values inaccurately predicted to be positive
False Negative (FN) (Type 2 Error)	positive values inaccurately predicted to be negative

• the word True or False says whether the model was right
• the word Positive or Negative says what the model predicted
• the diagonal of the matrix holds the correct predictions, so a good model concentrates its counts there

Metrics from the Matrix

Metric	Definition	Formula
Accuracy	the proportion of the total number of predictions that are correct	$\frac{TP+TN}{TP+TN+FP+FN}$
Precision (Positive Predictive Value)	the proportion of positive cases that are correctly identified	$\frac{TP}{TP+FP}$
Recall (Sensitivity)	the proportion of actual positive cases which are correctly identified	$\frac{TP}{TP+FN}$
Negative Predictive Value	the proportion of negative cases that are correctly identified	$\frac{TN}{TN+FN}$
Specificity	the proportion of actual negative cases which are correctly identified	$\frac{TN}{TN+FP}$
Rate	a measuring factor with 4 types, TPR, FPR, TNR and FNR	nil

Use Cases

• Accuracy gives an overall assessment of correctness, and it can be misleading on imbalanced datasets where 1 class significantly outnumbers the other
• Precision is used in cases where false positives are costly or undesirable
• Recall is used when missing a positive can have serious consequences

Code

from sklearn.metrics import accuracy_score, precision_score, recall_score, confusion_matrix
import seaborn as sns
 
def getConfMatrix(pred_data, actual):
    conf_mat = confusion_matrix(actual, pred_data, labels=[0,1])
    accuracy = accuracy_score(actual, pred_data)
    precision = precision_score(actual, pred_data, average='micro')
    recall = recall_score(actual, pred_data, average='micro')
    sns.heatmap(conf_mat, annot=True, fmt=".0f", annot_kws={"size": 18})
    print('Accuracy: ' + str(accuracy))
    print('Precision: ' + str(precision))
    print('Recall: ' + str(recall))

• confusion_matrix(actual, pred_data) takes the true labels first and the predictions second, so swapping them transposes the result
• labels=[0,1] fixes the order of the classes, which keeps the axes readable
• average='micro' selects how the score is aggregated across classes, which is explained in Micro and Macro Metrics
• seaborn draws the matrix as a heatmap, where annot=True prints the counts inside the cells