Part 3 · Deep Learning & AI

3.8 Robustness and adversarial examples

Why small changes cause big mistakes

10-minute readPrerequisites: 3.1 Neural networks, 3.2 Backpropagation▶ Video (about 5 min)

What you will learn

Explain robustness (decisions that do not change when the input changes a little), and the difference between natural changes and deliberate ones (adversarial examples)
Explain with a dot-product calculation why changes that are tiny per pixel add up to a big effect across many pixels
Explain how robustness is evaluated and countermeasures such as data augmentation, adversarial training and redundancy, in the context of automated driving

Build intuition

People recognize the same sign as the same sign even with a few raindrops on it, or in somewhat darker light. The property of a decision not changing when the input changes a little is called robustness. It matters for AI perception too — yet deep neural networks turn out to be surprisingly fragile at times.

The best-known example is the adversarial example. Changing every pixel of an image by an amount too small for the human eye to notice (around 1% of brightness) — but in a “chosen direction” — can make an AI confidently answer that it is something else. Experiments have been reported in which a few stickers on a stop sign made it be recognized as a speed-limit sign.

The key point here is that random changes and targeted changes are completely different. Changes of the same size can have almost no effect as random noise, yet drop accuracy to nearly zero when aligned with the worst direction.

Accuracy as the size of the change grows. For the same size, random noise and targeted changes (adversarial examples) have completely different effects. An adversarially trained model gives up a little accuracy on unchanged inputs in exchange for resisting targeted changes. The numbers are an illustrative example.

Check with math

① Writing robustness as a formula

Let the model be f, the input x and a slightly changed input x′. If the answer does not change whenever every pixel changes by at most ε, then f is robust around x:

$$\max_i\, |x'_i - x_i| \le \varepsilon \;\Longrightarrow\; f(x') = f(x)$$

ε (epsilon) is the size of change allowed (see Notation).

② Adding up small changes: why it is fragile

Take the neuron score of 3.1, $z = \mathbf{w} \cdot \mathbf{x} + b$. If each pixel $x_i$ is moved by ε in the same direction as the sign of its weight $w_i$, the score changes by

$$\Delta z = \mathbf{w} \cdot (\mathbf{x}' - \mathbf{x}) = \varepsilon \sum_{i} |w_i|$$

The point is that, although each effect is tiny, the effects of all the pixels line up in the same direction and add together. With 100,000 pixels, an average weight size of 0.01 and ε = 0.01 (1% of brightness), $\Delta z = 0.01 \times 100{,}000 \times 0.01 = 10$. An answer that was z = 4 (“pedestrian 0.98” after the sigmoid function) flips to z = 4 − 10 = −6 (“pedestrian 0.002”). The more pixels — the higher the input’s dimension — the bigger this sum becomes. Random noise does not do this, because its pluses and minuses cancel out.

③ How targeted changes are made

Real networks are not straight lines, but targeted changes can be made in the same spirit. Using the backpropagation of 3.2, compute the slope of the loss with respect to the input pixels instead of the weights, and move each pixel by ε in the direction that increases the loss:

$$\mathbf{x}' = \mathbf{x} + \varepsilon\, \mathrm{sign}\bigl(\nabla_{\mathbf{x}} L\bigr)$$

sign returns +1 for positive values and −1 for negative ones. This method, called FGSM, creates an adversarial example with a single slope computation.

④ Evaluation and countermeasures

Testing with added degradation: add rain, fog, snow, blur, noise, brightness changes and so on step by step, and examine how performance changes with the size of the change, as in the figure. Public evaluation datasets collecting natural corruptions are also available.
Searching for the worst case: besides random changes, use slopes to search for the worst change and check performance there too. For small models, there are methods that mathematically prove the answer cannot change within a given range.
Data augmentation: train on images with added rain, brightness changes and the like to withstand natural variation.
Adversarial training: create adversarial examples during training and learn to answer them correctly. This strengthens resistance to targeted changes, but often lowers accuracy on unchanged inputs a little, and takes more computation.

Summary

Robustness is the property of a decision not changing when the input changes a little. Deep networks can be badly fooled by targeted changes too small for people to see (adversarial examples)
One reason is addition in high dimensions: changes that are tiny per pixel add up to a large change in score when aligned. Targeted changes can be made using the slope with respect to the input
Evaluate with both random degradation and worst-case changes. Countermeasures include data augmentation, adversarial training, and combining different sensors and information

Quick quiz

Q1. In a model with 50,000 pixels and an average weight size of 0.01, each pixel is moved by ε = 0.02 in the direction of its weight’s sign. How much does the score z change?

$\Delta z = 0.02 \times 50{,}000 \times 0.01 = 10$. Even a slight 2% change per pixel is enough to flip the answer.

Q2. In a test with added random noise, accuracy barely dropped. Can you say the model is also resistant to adversarial examples?

No. With random noise, the per-pixel effects cancel out, so the impact is small. Targeted changes are aligned with the worst direction, so random tests say nothing about the worst case.

Q3. What are the main drawbacks of adversarial training?

It often lowers accuracy on ordinary, unchanged inputs a little, and it takes more computation, because adversarial examples must keep being created during training.

Q4. Against attacks that put stickers on signs, what defenses are there besides making the recognition model stronger?

Cross-checking against sign information registered in the map, comparing with the results of other sensors such as LiDAR, and checking that decisions do not change abruptly over time. The aim is that if one means is fooled, another notices.