Part 3 · Deep Learning & AI

3.11 Testing AI

How to check what has no single right answer

10-minute readPrerequisites: 3.10 Metrics for safety, 1.8 Statistics of rare events▶ Video (about 5 min)

What you will learn

Explain why a “correct answer” is hard to prepare when testing AI (the test oracle problem)
Explain the ideas of metamorphic testing, regression testing with negative flips, and slice testing
Calculate an upper bound when no violations are found, the number of cases that got worse, and weakness under particular conditions

Build intuition

For ordinary software, you can decide “this output is correct for this input” and test against it. Camera images, however, come in endless combinations of weather, lighting, angle and content, and people cannot label the correct answer (what is where) for all of them. The means of judging what is correct is called the test oracle, and how to provide one is the central problem of testing AI. So instead of the correct answer itself, three ideas are combined:

Check relations (metamorphic testing): if an image is flipped left to right, the detected boxes should flip too; if the brightness changes slightly, the detections should not change. Relations like these — how the output should change when the input is changed — can be checked without a correct answer.
Compare with the previous version (regression testing): check on the same test data that a new model has not lost what the previous model could do.
Split by condition and search (slice testing): divide the test data by condition — night, rain, long range, children — and look for weak spots.

An old and a new model compared on the same 10,000 cases (illustrative numbers). Overall accuracy went up, but the new model gets wrong 200 cases the old model got right.

Check with math

① Metamorphic testing: an upper bound when there are no violations

Flip 5,000 images left to right and check whether the detected boxes flip too. No labels are needed, so the number of cases can be increased as far as there are images. If there is not a single violation, the rule of three from 1.8 gives the upper bound on the violation rate (95% confidence):

$$\frac{3}{n} = \frac{3}{5000} = 0.06\%$$

A violation, when found, is itself a clue that the model is relying on something it should not (the shortcut learning of 3.9). But a relation being kept does not mean the answer itself is correct: if the model misses a pedestrian in the same way before and after flipping, there is no violation. Metamorphic testing supplements tests with correct answers; it does not replace them.

② Regression testing: count what got worse even when the average improves

As in the figure, suppose that on 10,000 cases the old model is 95.0% accurate (9,500 cases) and the new model 96.0% (9,600). If the new model gets wrong 200 of the cases the old model got right, and fixes 300 of those the old model got wrong, then

$$9500 - 200 + 300 = 9600$$

and the average has risen by 1 point. These 200 cases (2.0%) are negative flips. To users, they are “things that worked before and stopped working after the update”, and if they include dangerous situations (such as nearby pedestrians), safety may go down even though the average goes up. Regression testing therefore checks the number and content of negative flips as well as comparing averages (the paired comparison of 3.10).

③ Slice testing: enough cases per condition to tell weakness apart

For a model with an overall miss rate of 2%, check whether it is weak specifically in “rain at night”. Test n images of that condition and suppose x misses are found. If the rate were the overall 2%, about 0.02n misses would be expected. Treating the number of misses as Poisson (1.3), the size of the deviation is

$$z = \frac{x - 0.02\,n}{\sqrt{0.02\,n}}$$

If the true miss rate in this condition is 6% (three times the overall rate), then with n = 50 images about 3 misses are expected against a baseline of 1, giving z = (3 − 1) / 1 = 2 — right on the edge of telling them apart. With n = 200, about 12 misses against a baseline of 4 give z = (12 − 4) / 2 = 4, clearly distinguishable. Without enough cases per condition, weak spots stay buried in the average (5.9).

Summary

In testing AI, a correct answer cannot be prepared for every input (the test oracle problem). Metamorphic testing, which checks relations between inputs and outputs, fills the gap; with zero violations, the rule of three gives an upper bound
For updates, check not only the average but also the number and content of cases that stopped working (negative flips)
Slice testing splits the test data by condition to find weak spots. Telling them apart needs enough cases per condition

Quick quiz

Q1. On 30,000 images with altered brightness, not a single detection changed (no violations). What is the upper bound on the violation rate (95% confidence)?

3 / 30,000 = 0.01%.

Q2. On 10,000 cases, the old model got 9,000 right, the new model 9,200, with 150 negative flips. How many cases did the new model fix?

From 9,000 − 150 + x = 9,200, x = 350.

Q3. A model has an overall miss rate of 2%. On 400 images with glare, it missed 20. Is it weak in glare?

The expected count is 0.02 × 400 = 8, so z = (20 − 8) / √8 ≈ 4.2. That is well above 2, so it is weak in glare.

Q4. “Neuron coverage on the test data reached 100%, so testing is sufficient.” What is wrong with this view?

Neuron coverage only counts which parts of the network were activated; it says nothing about whether dangerous situations (night, rain, nearby pedestrians and so on) were tested. Coverage has to be checked against ODD conditions and scenarios, and per-condition performance and negative flips also need to be examined.