Part 5 · SOTIF & Validation

5.9 Safety of AI components

ISO/PAS 8800 and dataset quality

10-minute readPrerequisites: 5.3 Functional insufficiencies and triggering conditions, 2.8 Data splits and distribution shift▶ Video coming soon

What you will learn

Explain the flow set out in ISO/PAS 8800 for using AI components safely (requirements, data, verification, monitoring in operation)
Explain the aspects of dataset quality (coverage, correctness, independence, representativeness, traceability) and calculate how much test data each condition needs
Confirm by calculation how label errors and data gaps make apparent performance diverge from real danger

Build intuition

As we saw in Part 3, learned AI components make mistakes — with nothing broken — in situations absent from their training data, or because of small differences in appearance. These are exactly the functional insufficiencies of 5.1. ISO/PAS 8800 (2024) sets out an approach to using such AI components safely, supplementing functional safety (ISO 26262) and SOTIF (ISO 21448) for AI. In summary:

Derive requirements for the AI component (under which conditions, and with what performance) from the system’s safety requirements
Set requirements for the datasets used for training, evaluation and testing as well, check their quality and manage their versions
Verify performance by condition, robustness, uncertainty and so on
Keep monitoring in operation, for unfamiliar inputs and the like
Continue retraining and re-verifying with data from after release

For a pedestrian-detection component, for example, the requirements might read like this (examples):

Aspect	Example requirement	How to check
Performance	Detect at least 99% of pedestrians within 60 m inside the ODD	Detection rate by condition with confidence intervals (5.3)
Robustness	Performance does not drop sharply with rain, low sun or a dirty lens	Tests with added corruptions (3.8)
Uncertainty	Confidences are calibrated	Reliability diagrams and ECE (3.7)
Out of distribution	Notice and report unfamiliar inputs	Out-of-distribution test data (2.8)
Timing	Deliver results within the allotted time	Measuring processing time (4.2)

Because an AI component’s behaviour is determined by data, dataset quality is central to safety:

Aspect	Meaning	Example
Coverage	Covers the ODD’s conditions sufficiently	Are there too few images of foggy nights?
Correctness	Few errors in the labels (ground truth)	Missed pedestrians, misaligned boxes
Independence	Training, evaluation and test data are not mixed	Images from the same drive do not appear in both (2.8)
Representativeness	Reflects the proportions of real situations	No bias such as all-urban data with no rural roads
Traceability	Where the data came from and how it was processed is known	Version control, records of labelling procedures

An example count of training images for each combination of ODD conditions (illustrative numbers). Red cells fall short of the set minimum (1,000 images) and form data gaps.

The figure counts training images by ODD condition (weather × lighting). Even when the total number is large, looking at combinations of conditions reveals cells that fall short.

Check with math

① How much test data is needed?

Showing “a detection rate of at least 99% under this condition” means showing that the probability of a miss is at most 1%, so the rule of three from 1.8 applies directly. With zero misses, the number of pedestrians needed is

$$n \ge \frac{3}{1 - 0.99} = 300$$

(with one miss, 4.7 ÷ 0.01 = 470). This is needed for each cell of conditions. Showing 99.9% in each of the 12 cells in the figure needs 3,000 per cell — 36,000 pedestrians in all. And since test data must not be used for training, it is collected separately from the training data.

② Label errors hide the weak spots

The ground truth (labels) of evaluation data contains errors too. The trouble is that the pedestrians people tend to miss when labelling — small, occluded, dark — are the very ones AI tends to miss. Suppose there are really 10,000 pedestrians and the AI detects 9,700 (a true detection rate of 97%). If the labellers missed 200 difficult pedestrians and the AI also missed 150 of those 200, then of the 9,800 labelled pedestrians the AI missed only 150, so

$$\text{apparent detection rate} = \frac{9800 - 150}{9800} \approx 98.5\%$$

— higher than the truth. Label errors hide precisely the weak spots. For safety-related test data, correctness is checked by having several people label the same data and examining disagreements, re-reviewing difficult situations with particular care, and so on.

③ Data gaps are hard to see in averages

Of the cells in the figure, 6 of 12 (50%) meet the minimum, but weighted by driving time the covered cells account for 89% and the gap cells only 11%. The overall miss rate is, as in 2.8, the average of the miss rates by condition $m_c$ weighted by the share of driving time $w_c$:

$$m = \sum_c w_c\, m_c$$

If the covered cells have a miss rate of 1% and the gap cells 10%, then m = 0.89 × 0.01 + 0.11 × 0.10 ≈ 0.020 — more than half of all misses come from gap cells that make up only 11% of the time (illustrative numbers). Data gaps can become exactly where danger concentrates. The choice is then between adding data and taking those conditions out of the ODD.

Summary

Most errors of AI components are functional insufficiencies that occur with nothing broken. ISO/PAS 8800 sets out a flow of deriving requirements for AI components and datasets from system requirements, verifying them, and monitoring in operation
Showing 99% per condition takes 3/(1 − 0.99) = 300 test cases per cell. Label errors hide the AI’s weak spots and inflate apparent performance
Data gaps can look small as a share of time yet produce most of the misses. Manage data for coverage, correctness, independence, representativeness and traceability

Quick quiz

Q1. To show a detection rate of at least 99.5% under one condition with zero misses, how many pedestrians of test data are needed?

3 ÷ (1 − 0.995) = 3 ÷ 0.005 = 600.

Q2. In ②, suppose that of 10,000 real pedestrians the AI detects 9,600, and of the 300 the labellers missed, the AI also missed 200. What is the apparent detection rate?

Of the 9,700 labelled pedestrians, the AI missed 400 − 200 = 200, so (9,700 − 200) ÷ 9,700 ≈ 97.9% — higher than the true 96%.

Q3. In ③, if the covered cells have a miss rate of 2% and the gap cells 20%, what is the overall miss rate? (Weights 0.89 and 0.11.)

0.89 × 0.02 + 0.11 × 0.20 = 0.0178 + 0.022 ≈ 0.040, about 4%.

Q4. “We have collected a million training images, so we have enough data.” What is wrong with this view?

However many images there are, there may be gaps when looked at by combination of ODD conditions (coverage). Label errors (correctness), mixing of training and test data (independence) and differences from real-world proportions (representativeness) also need checking.