Source

Adapted from my own handwritten and hand-typed notes from Year 1 to Year 3 of my EEE degree

A confusion matrix counts how a classifier's predictions compare with the actual labels. Rows are actual values, columns are predicted values.

Layout

Predicted False (0)Predicted True (1)
Actual False (0)TNFP
Actual True (1)FNTP

The four outcomes:

  • TP (true positive): True predicted as True
  • TN (true negative): False predicted as False
  • FP (false positive): False predicted as True, a false alarm
  • FN (false negative): True predicted as False, a miss

Metrics

Let (actual positives) and (actual negatives).

MetricFormulaAlso called
Accuracyfraction correct
True positive raterecall, sensitivity, hit rate
True negative ratespecificity
False positive ratefalse alarm rate
False negative ratemiss rate
Precisionshare of predicted True that are right

Example: Legendary from Total

DataTNFPFNTPAccuracyFPRFNR
Train (600)55252419
Test (200)1780139

Accuracy looks high, but the FNR is over 50%: most legendary Pokemon are missed. This happens because the classes are very imbalanced (only about 7% are legendary), so predicting False almost always is already accurate.

Which Error to Reduce

Reducing both errors together is often not possible. The problem decides which matters more:

  • for disease screening a false negative (missed patient) is worse
  • for spam filtering a false positive (lost real email) is worse

Code

from sklearn.metrics import confusion_matrix
 
y_train_pred = dectree.predict(X_train)
print('Accuracy train:', dectree.score(X_train, y_train))
cm = confusion_matrix(y_train, y_train_pred)
sb.heatmap(cm, annot=True, fmt='.0f', annot_kws={'size': 18})
 
tn, fp, fn, tp = cm.ravel()
print('TPR', tp / (tp + fn), 'FPR', fp / (fp + tn))

These numbers come from the tree in Decision Tree Classification.