Part 3 · Deep Learning & AI

3.9 Explainability (XAI)

Exploring why a model decided what it did

10-minute readPrerequisites: 3.3 CNNs and image recognition, 3.8 Robustness and adversarial examples (input slopes)▶ Video (about 6 min)

What you will learn

Explain the purpose of explainability (XAI) and the kinds of explanation (interpretable models vs after-the-fact explanations; single decisions vs overall tendencies)
Work through small examples of saliency maps, occlusion, feature attributions (SHAP) and counterfactuals
Explain the limits of explanations (plausible but wrong) and how they are used in the safety argument for automated driving

Build intuition

In 3.3 we met a model that identified wolves by the snow in the background. Even with high accuracy, whether a model decides for the right reasons cannot be told from accuracy alone. Techniques that examine which parts of the input an AI’s decision relies on are called explainability, or XAI (explainable AI).

Consider an example that could arise in automated driving. A pedestrian detector scored high accuracy in testing. But examining where its decisions came from showed that it responded strongly not to the shape of people but to crosswalk stripes. Because almost all the pedestrians in its training data were on crosswalks, it had learned the shortcut “where there are stripes, there is a pedestrian”. As it stands, it could miss people crossing where there is no crosswalk.

The places that mattered to each decision, overlaid by depth of color (schematic). The left detector responds to the person’s shape; the right one to the stripes at their feet. Accuracy alone cannot tell them apart, but explanation techniques reveal the difference.

There are several kinds of explanation:

Method	What it reveals	Example
Use an interpretable model	The weights or rules themselves	The weights of 2.2 and 2.4, a small decision tree from 2.5
Saliency map	How much the answer changes when each pixel changes a little	The pixels on the pedestrian’s outline light up
Occlusion (hiding parts)	How much the confidence drops when part of the input is hidden	Hiding the head lowers it a lot
Feature attribution (SHAP, etc.)	How much each input pushed the answer up or down	“The lateral speed pushed the cut-in prediction up”
Counterfactual	How the input would have to change to change the answer	“0.2 m/s slower and it would have predicted no cut-in”

An explanation of a single decision is a local explanation; one about the model’s overall tendencies is a global explanation. The stripe example above is a global tendency that only becomes visible once explanations of many images are gathered.

Check with math

① Saliency maps: the slope with respect to the input

With y the model’s output (say, the “pedestrian” score), the salience of each pixel $x_i$ can be expressed as the rate at which y changes when that pixel changes a little — the input slope used in 3.8:

$$s_i = \left| \frac{\partial y}{\partial x_i} \right|$$

For the straight-line neuron of 3.1, $y = \mathbf{w} \cdot \mathbf{x} + b$, this gives $s_i = |w_i|$: the larger a pixel’s weight, the more salient it is. For deep networks too, one pass of the backpropagation of 3.2 gives the salience of every pixel.

② Occlusion: hide, and watch the confidence drop

Cover part of the image with grey and see how much the confidence drops:

$$\Delta_k = f(\mathbf{x}) - f(\mathbf{x}\ \text{with region}\ k\ \text{hidden})$$

Region hidden	Left detector’s confidence	Right detector’s confidence
(nothing hidden)	0.95	0.95
Head and body	0.30 (down 0.65)	0.85 (down 0.10)
Stripes at the feet	0.92 (down 0.03)	0.25 (down 0.70)
Sky in the background	0.94 (down 0.01)	0.94 (down 0.01)

The right detector barely loses confidence when the person is hidden, but loses a great deal when the stripes are hidden — clear evidence that it is looking at the stripes, not the person (the numbers are an illustrative example).

③ Feature attribution: how far each input pushed from the average

Take the cut-in prediction of 2.4 and 2.5 with the following model, using lateral speed v (m/s) and turn signal s (1 if on):

$$z = 6.0\,v + 1.5\,s - 3.5, \qquad p = \sigma(z)$$

If the averages over the data are $\bar{v} = 0.4$ and $\bar{s} = 0.4$, an average situation scores $z = -0.5$ (a 38% chance of cutting in). A car with v = 0.8 and s = 1 scores z = 2.8 (94%). Splitting the difference of 3.3 by input gives

$$\underbrace{6.0 \times (0.8 - 0.4)}_{\text{speed}\ +2.4} + \underbrace{1.5 \times (1 - 0.4)}_{\text{signal}\ +0.9} = 3.3$$

so we can say “the lateral speed pushed the cut-in prediction up by 2.4 and the turn signal by 0.9”. The idea of fairly dividing the difference from an average situation among the inputs is SHAP, and for a straight-line model like this one it takes exactly this form. For deep models, the same idea is computed approximately.

④ The limits of explanations

Plausible is not the same as correct: some saliency-map methods have been reported to produce similarly “convincing” pictures even for untrained networks with random weights. The explanation methods themselves need checking.
An explanation is a “model of the model”: it simplifies a complex model into something people can understand. Something is lost in the simplification, and what the explanation shows can differ from how the decision was actually made. The attention weights of 3.5 call for the same caution.
People believe explanations that make sense to them: when the expected picture appears, people tend to stop looking. Explanations do not replace testing and data checks.

Summary

Explainability (XAI) examines which parts of the input an AI’s decisions rely on. It checks the “reasons” that accuracy cannot reveal, and helps find shortcuts and the causes of errors
Methods include saliency maps (slopes with respect to the input), occlusion (hiding parts and watching the confidence drop), feature attribution (SHAP: dividing the difference from average among the inputs) and counterfactuals
An explanation is a “model of the model” that can be plausible but wrong. Combine it with testing and data checks, and use it as one ingredient of the safety argument

Quick quiz

Q1. For a pedestrian detection at confidence 0.90, hiding the head gave 0.35 and hiding the sky in the background gave 0.89. Which mattered to the decision?

The head. Hiding it dropped the confidence by 0.55, while the sky (0.01) barely mattered.

Q2. With the model in the text (z = 6.0v + 1.5s − 3.5, averages v̄ = 0.4 and s̄ = 0.4), what are the contributions of speed and turn signal for a car with v = 0.2 and s = 1?

Speed: 6.0 × (0.2 − 0.4) = −1.2 (pushed the prediction down); turn signal: 1.5 × (1 − 0.4) = +0.9 (pushed it up). The total, −0.3, matches the difference between this car’s score, −0.8, and the average situation’s −0.5.

Q3. A saliency map produced a “convincing” picture that lined up neatly with the pedestrian’s shape, so the team concluded that “the model decides for the right reasons”. What is the problem?

Some explanation methods produce similar pictures even for untrained networks, so a convincing picture does not prove correctness. Check with other methods such as occlusion, and combine with testing and data checks.

Q4. A pedestrian detector turns out to rely on crosswalk stripes. In which situations is this dangerous? What should be done?

It may miss pedestrians crossing where there is no crosswalk, or on roads where the stripes have faded. Collect more data of pedestrians away from crosswalks, retrain, and check such situations in testing.