A precision-recall curve shows how precision and recall change as the decision threshold of a classifier moves. It is the better choice than the ROC curve when the positive class is rare. It complements Area Under Curve (AUC-ROC).

Building The Curve

  • the model gives a score for each sample
  • for every threshold, label samples above it as positive
  • compute precision and recall at that threshold, and plot recall on the axis and precision on the axis
  • lowering the threshold usually raises recall and lowers precision

Why PR Beats ROC On Rare Positives

  • the ROC curve uses the false positive rate,
  • when negatives are many, is huge, so stays small even when false positives outnumber true positives
  • precision has no in it, so it shows directly how many alarms are real
  • example: finding a rare signal in hours of background noise, where almost every window is negative

Summary Numbers

  • average precision (AP), close to the area under the PR curve
  • a random classifier's PR curve is a flat line at the positive rate, not at 0.5 as with ROC

Choosing A Threshold

  • fix the lowest precision (or the highest false alarm rate) that users accept, then choose the threshold with the highest recall
  • or maximise when both errors cost the same
  • the default threshold of 0.5 is rarely the best one, especially with class imbalance or label smoothing