Source
Adapted from my own handwritten and hand-typed notes from Year 1 to Year 3 of my EEE degree
A confusion matrix counts how a classifier's predictions compare with the actual labels. Rows are actual values, columns are predicted values.
Layout
| Predicted False (0) | Predicted True (1) | |
|---|---|---|
| Actual False (0) | TN | FP |
| Actual True (1) | FN | TP |
The four outcomes:
- TP (true positive): True predicted as True
- TN (true negative): False predicted as False
- FP (false positive): False predicted as True, a false alarm
- FN (false negative): True predicted as False, a miss
Metrics
Let (actual positives) and (actual negatives).
| Metric | Formula | Also called |
|---|---|---|
| Accuracy | fraction correct | |
| True positive rate | recall, sensitivity, hit rate | |
| True negative rate | specificity | |
| False positive rate | false alarm rate | |
| False negative rate | miss rate | |
| Precision | share of predicted True that are right |
Example: Legendary from Total
| Data | TN | FP | FN | TP | Accuracy | FPR | FNR |
|---|---|---|---|---|---|---|---|
| Train (600) | 552 | 5 | 24 | 19 | |||
| Test (200) | 178 | 0 | 13 | 9 |
Accuracy looks high, but the FNR is over 50%: most legendary Pokemon are missed. This happens because the classes are very imbalanced (only about 7% are legendary), so predicting False almost always is already accurate.
Which Error to Reduce
Reducing both errors together is often not possible. The problem decides which matters more:
- for disease screening a false negative (missed patient) is worse
- for spam filtering a false positive (lost real email) is worse
Code
from sklearn.metrics import confusion_matrix
y_train_pred = dectree.predict(X_train)
print('Accuracy train:', dectree.score(X_train, y_train))
cm = confusion_matrix(y_train, y_train_pred)
sb.heatmap(cm, annot=True, fmt='.0f', annot_kws={'size': 18})
tn, fp, fn, tp = cm.ravel()
print('TPR', tp / (tp + fn), 'FPR', fp / (fp + tn))These numbers come from the tree in Decision Tree Classification.