Part 3 · Deep Learning & AI

3.7 Uncertainty and calibration

Can AI report its own confidence correctly?

10-minute readPrerequisites: 2.4 Logistic regression, 2.7 Evaluation metrics▶ Video (about 5 min)

What you will learn

Check calibration — whether confidence matches actual accuracy — with reliability diagrams and ECE
Explain how temperature scaling fixes overconfidence
Explain the difference between aleatoric and epistemic uncertainty, and how uncertainty is used, and where it falls short, in automated driving

Build intuition

When a detector reports “pedestrian 0.95”, can you trust that 0.95? Think of a weather forecast. If, on all the days with a “70% chance of rain”, it actually rained on exactly 70% of them, the forecast’s numbers can be trusted (1.1). The same goes for AI: if 95% of the cases where it answered “0.95” really were pedestrians, its confidence is reported correctly. The model is then said to be well calibrated.

Deep neural networks, however, are known to be prone to overconfidence: of the cases answered with “0.95”, only about 80% may actually be correct. The figure below is an example of a reliability diagram, which checks the actual accuracy at each level of confidence. Points below the ideal dotted line mean overconfidence.

An example reliability diagram. Detections are grouped by confidence, and each group’s actual accuracy is plotted as a point. Before calibration, actual accuracy falls short at every confidence level (overconfidence). After temperature scaling makes the confidence more modest, the points lie almost on the ideal dotted line. The numbers are an illustrative example.

Another important question is why the model is unsure. Uncertainty that comes from the data itself being ambiguous — a distant figure in the dark that nobody could classify from the image alone — is called aleatoric uncertainty. More data does not reduce it. Uncertainty that comes from the model’s lack of knowledge — an unusual vehicle absent from the training data — is called epistemic uncertainty. This kind can be reduced with more data.

Check with math

① What calibration means

With the model’s answer $\hat{Y}$, its confidence $\hat{p}$ and the true answer Y, being well calibrated means:

$$P(Y = \hat{Y} \mid \hat{p} = p) = p$$

That is, “when the model answers with confidence p, it is actually right with probability exactly p” (the conditional probability of 1.2). High accuracy and good calibration are separate properties.

② ECE: the gap as a single number

Group detections into confidence bins b, and take the average, weighted by the counts $n_b$, of the gap between each bin’s average confidence $\mathrm{conf}_b$ and its actual accuracy $\mathrm{acc}_b$. This is the ECE (expected calibration error).

$$\mathrm{ECE} = \sum_{b} \frac{n_b}{N}\,\bigl|\,\mathrm{acc}_b - \mathrm{conf}_b\,\bigr|$$

Confidence bin	0.5–0.6	0.6–0.7	0.7–0.8	0.8–0.9	0.9–1.0
Count	100	150	200	250	300
Average confidence	0.55	0.65	0.75	0.85	0.95
Actual accuracy	0.50	0.55	0.62	0.70	0.80
Confidence after temperature scaling	0.53	0.58	0.63	0.70	0.81

Before calibration, ECE = (100 × 0.05 + 150 × 0.10 + 200 × 0.13 + 250 × 0.15 + 300 × 0.15) ÷ 1000 ≈ 0.13 — on average, the model is 13 points overconfident.

③ Temperature scaling: fixing overconfidence

As with the temperature in 3.6, divide the score z by a temperature T before turning it into a probability. For a choice between two answers, $\hat{p}' = \sigma(z / T)$ (σ is the sigmoid function, 2.4). T is chosen to make ECE small on validation data not used for training (2.6). In this example T = 2, which makes the confidence more modest, as in the last row of the table, and lowers ECE to about 0.01.

Dividing by a temperature does not change which answer has the highest probability, so accuracy stays the same. Only the way confidence is reported changes.

④ Measuring epistemic uncertainty: disagreement between answers

Epistemic uncertainty is hard to see from one model’s confidence alone. So we compare the answers of several separately trained models (the ensembles of 2.5). In a familiar scene, five models might give “pedestrian” probabilities of 0.92, 0.94, 0.91, 0.93 and 0.95 — they agree, with a spread (standard deviation) of about 0.01. In an unfamiliar scene they might give 0.95, 0.30, 0.70, 0.10 and 0.85 — all over the place, with a standard deviation of about 0.33. Disagreement is a clue that “these models do not know this scene”. The out-of-distribution detection of 2.8 serves the same purpose.

Summary

In a well-calibrated model, cases answered with confidence p are correct exactly p of the time. Check with reliability diagrams and ECE; deep networks tend to be overconfident
Temperature scaling divides the scores by a temperature to make confidence more modest. Accuracy is unchanged; only the way confidence is reported changes
Uncertainty comes from ambiguity in the data itself (aleatoric) and from a lack of knowledge (epistemic). The latter is measured through ensemble disagreement and similar methods, and linked to safe-side behavior

Quick quiz

Q1. There were 200 detections with confidence 0.8, and 120 of them were actually correct. Is the model well calibrated?

No. Actual accuracy is 120 ÷ 200 = 60%, 20 points below the confidence of 0.8: the model is overconfident.

Q2. There are 500 detections at confidence 0.9 (actual accuracy 0.85) and 500 at confidence 0.6 (actual accuracy 0.62). What is the ECE?

(500 × 0.05 + 500 × 0.02) ÷ 1000 = 0.035.

Q3. For a distant figure in the dark, nobody can tell from the image alone whether it is a person or a sign. Which kind of uncertainty is this? Does more data reduce it?

Aleatoric uncertainty. The image itself is ambiguous, so more data of the same kind does not reduce it. Other information is needed — getting closer, or using another sensor such as LiDAR.

Q4. For an unusual vehicle absent from the training data, five models gave widely different answers. Which kind of uncertainty is this? How can it be reduced?

Epistemic uncertainty: the models do not know this vehicle. Collecting data on that kind of vehicle and training on it reduces the uncertainty. Until then, the large uncertainty should trigger safe-side behavior.