2.7 Evaluation metrics
Confusion matrix, precision, recall and ROC
What you will learn
- Calculate recall, precision and related metrics from the four cells of a confusion matrix (true positives, misses, false alarms, true negatives)
- Explain, in terms of minority classes and how rare the target is, why “high accuracy” alone says little about safety
- Explain the trade-off between misses and false alarms as the threshold moves, using the ROC curve and AUC
Build intuition
A pedestrian detector was tested on 1,000 candidates. Only 100 of them were actually pedestrians; the other 900 were signs, poles and other non-pedestrians. For each candidate the detector outputs a “pedestrian-likeness” score, and anything above the threshold is classified as a pedestrian.
The result: “91% accuracy”. That sounds impressive — yet a useless detector that always answers “no pedestrian” would also score 90%, simply because pedestrians are in the minority. Safety-related evaluation starts by counting the results in four separate cells. This table is called the confusion matrix.
| Classified: pedestrian | Classified: not a pedestrian | |
|---|---|---|
| Actually a pedestrian (100) | 69 (correct detection: TP) | 31 (miss: FN) |
| Actually not a pedestrian (900) | 60 (false alarm: FP) | 840 (correct rejection: TN) |
Misses and false alarms happen because the score distributions of pedestrians and non-pedestrians overlap. Wherever the threshold is placed, both kinds of error cannot be driven to zero at the same time.
Check with math
① The main metrics from the confusion matrix
| Metric | Formula | This example | Meaning |
|---|---|---|---|
| Recall (detection rate) | \(\dfrac{\mathrm{TP}}{\mathrm{TP} + \mathrm{FN}}\) | 69 ÷ 100 = 69% | The share of real pedestrians that were found |
| Miss rate | \(\dfrac{\mathrm{FN}}{\mathrm{TP} + \mathrm{FN}}\) | 31 ÷ 100 = 31% | The share of real pedestrians that were missed (= 1 − recall) |
| Precision | \(\dfrac{\mathrm{TP}}{\mathrm{TP} + \mathrm{FP}}\) | 69 ÷ 129 ≈ 53% | The share of “pedestrian” classifications that were real |
| False positive rate | \(\dfrac{\mathrm{FP}}{\mathrm{FP} + \mathrm{TN}}\) | 60 ÷ 900 ≈ 6.7% | The share of non-pedestrians wrongly classified as pedestrians |
| Accuracy | \(\dfrac{\mathrm{TP} + \mathrm{TN}}{N}\) | 909 ÷ 1000 ≈ 91% | The share of all N cases classified correctly |
Higher recall means fewer misses; higher precision means fewer false alarms. To combine the two into one number, use the F1 score (P is precision, R is recall):
The F1 score does not grow when only one of the two is high.
Metrics that share a denominator add up to 1. Recall and miss rate both have real pedestrians (TP + FN) as the denominator, so a recall of 99% and a miss rate of 1% say the same thing. Safety discussions usually use the miss rate: “99% versus 99.9% recall” looks like a small difference, but as miss rates, “1% versus 0.1%”, it is tenfold. Likewise, for the false positive rate, whose denominator is the non-pedestrians (FP + TN), 1 − false positive rate is called the specificity (true negative rate), a term common in medicine and statistics. Only precision has a different denominator, “the number of detections”, so it changes with how rare the target is, as ② shows.
Papers and standards often use other names and abbreviations: recall = sensitivity = true positive rate (TPR); miss rate = false negative rate (FNR); false positive rate (FPR), also called fall-out; specificity = true negative rate (TNR); precision = positive predictive value (PPV). Note that in object detection, which searches images for objects (3.4), the “non-pedestrians” cannot be counted, so TN is undefined and neither the false positive rate nor specificity can be calculated. False alarms are therefore usually expressed as counts per image, per km or per hour (such as the frequency of unnecessary emergency braking).
② Precision depends on how rare the target is
What happens if the same detector, with the same threshold, is used somewhere with only 10 pedestrians in every 1,000 candidates? It finds about 6.9 pedestrians and raises about 66 false alarms, so precision falls to about 9.5%. Recall and the false positive rate have not changed, yet fewer than one in ten “pedestrian” classifications is real. This is Bayes’ theorem from 1.2 at work. When comparing precision, make sure it was measured on data with the same mix.
③ Moving the threshold
| Threshold | 0.5 | 1.0 | 1.5 | 2.0 | 2.5 |
|---|---|---|---|---|---|
| Recall | 93% | 84% | 69% | 50% | 31% |
| Precision | 25% | 37% | 53% | 71% | 84% |
| Misses | 7 | 16 | 31 | 50 | 69 |
| False alarms | 278 | 143 | 60 | 20 | 6 |
Lowering the threshold reduces misses and increases false alarms (2.4); raising it does the opposite. The tables so far used a threshold of 1.5.
④ ROC curves and AUC
Plotting the false positive rate horizontally and recall vertically for every threshold, and joining the points, gives the ROC curve. The closer it comes to the top-left corner, the better the detector — few misses and few false alarms.
The area under the curve is called the AUC. It is “the probability that, picking one pedestrian and one non-pedestrian at random, the pedestrian gets the higher score” — about 0.92 for this detector, and 0.5 for random guessing. Because AUC does not depend on the threshold or on the share of pedestrians, it is handy for comparing detectors. But in the vehicle, a single threshold is actually used, and safety is judged on the misses and false alarms at that threshold.
Advanced: metrics have a range too
The metrics so far are “estimates” calculated from test data. With few test cases, their values move a lot through chance alone. If a test with 200 pedestrians gives a recall of 95% (190 detected), the standard error from 1.6 is
so the 95% confidence interval is roughly 92–98%. Even if another model scores 96% on the same 200 pedestrians, that is within this range, so we cannot say it is better (use the paired comparison of 3.10 to compare). Narrowing the range to ±1% takes about 4 × 0.95 × 0.05 ÷ 0.01² ≈ 1,900 pedestrians.
When there are no misses at all, use the rule of three from 1.8 instead of the interval formula. Zero misses among 300 pedestrians gives an upper bound on the miss rate of 3 / 300 = 1% — all we can say is “recall is at least 99%”. Precision is the same: its denominator is the number of alarms, so with few alarms the range is wide. Report metrics as a set: estimate, range (interval) and number of cases.
Connection to automated driving
- Do not judge by accuracy alone: the objects that lead to hazards — pedestrians, obstacles — are usually in the minority. Look at recall (misses) and false alarms for each type of object, not at accuracy.
- Set requirements on both misses and false alarms: misses lead to collisions; false alarms lead to unnecessary hard braking (with the risk of being rear-ended, and loss of occupants’ trust). Express false alarms as rates too — “per hour”, “per 10,000 km” — and set targets for both (1.8). → 5.5 Acceptance criteria and validation targets
- Measure separately by situation and by object: even with high overall recall, low recall at night, in rain, for children or for distant pedestrians is dangerous. Measure each condition separately and, where the counts are small, attach confidence intervals (1.6).
- Object detection also scores location: in object detection, which also locates each object in the image, a detection counts as correct only when the predicted box and the true box overlap enough (IoU above a set value). Evaluating by distance and object size matters too. → 4.3 Sensors and fusion
Summary
- Start evaluation by counting with a confusion matrix (TP, FN, FP, TN). Accuracy alone can make a model that misses rare, dangerous objects look good
- Recall measures how few misses there are; precision measures how few false alarms. Precision changes greatly with how rare the target is
- The threshold trades misses against false alarms. Compare detectors with ROC curves and AUC, and choose the operating threshold from the safety requirements
Quick quiz
Q1. A detector found 90 of 100 pedestrians and raised 30 false alarms. What are its recall and precision?
Recall is 90 ÷ 100 = 90%; precision is 90 ÷ (90 + 30) = 75%.
Q2. In data where only 1% of cases are pedestrians, what are the accuracy and recall of a model that always answers “no pedestrian”?
Accuracy is 99%, but recall is 0% — it never finds a single pedestrian. When the minority matters, accuracy cannot be trusted.
Q3. What usually happens to recall and precision when the threshold is raised?
Fewer cases are classified as pedestrians, so recall falls (more misses) and precision rises (fewer false alarms).
Q4. The same detector was evaluated in a busy city center and in the suburbs, where pedestrians are few. Recall was almost the same, but precision was much lower in the suburbs. Why?
In the suburbs pedestrians are rare, so non-pedestrian candidates make up a larger share. Even with the same false positive rate, false alarms outnumber real pedestrians by more, and precision falls (Bayes’ theorem from 1.2).
Further reading
- Next lesson: 2.8 Data splits and distribution shift
- Related: 1.2 Conditional probability and Bayes’ theorem, 1.6 Estimation and confidence intervals, 2.4 Logistic regression (thresholds)
- Glossary: Confusion matrix / accuracy, Precision / recall, False positive / false negative, F1 score, ROC curve / AUC, IoU
- Video lecture: YouTube (about 5 minutes) (the key points explained with slides and narration; the Japanese article has a Japanese video)