Part 5 · SOTIF & Validation

5.2 The four areas

Known / unknown × safe / hazardous

10-minute readPrerequisites: 5.1 What is SOTIF?, 1.8 Statistics of rare events▶ Video (about 5 min)

What you will learn

Explain the four areas of SOTIF (known safe, known hazardous, unknown hazardous, unknown safe) and the activities that move situations between them
Confirm by calculation that test driving finds common hazards but tends to leave rare ones
Explain the idea of estimating the share of kinds not yet seen (Good–Turing) and of residual risk that includes the unknown

Build intuition

Sort every situation an automated vehicle might meet by two questions. First: “do we know about this situation?” (known or unknown). Second: “does the system behave safely in it?” (safe or hazardous). Combining them gives four areas.

The four areas of SOTIF (schematic). Unknown hazards (area 3) are found through analysis, testing and searching, becoming known hazards (area 2), which are then fixed to become known safe (area 1).

Area 1 (known safe): situations that are well understood and confirmed to be safe. We want this area to grow
Area 2 (known hazardous): situations known to be dangerous. Measures (improving the function or performance, restricting the ODD, changing how the driver is informed) move them to area 1 or make the remaining danger small enough
Area 3 (unknown hazardous): dangerous situations nobody has noticed yet. The most worrying area; analysis, testing and searching find them and move them to area 2
Area 4 (unknown safe): situations nobody has thought about that are in fact safe. Confirming them and moving them to area 1 avoids narrowing the ODD more than necessary

The goal of SOTIF work is to make areas 2 and 3 sufficiently small. Area 2 can be listed and dealt with item by item, but area 3 is “what we do not know”, so how much of it remains cannot be counted directly. Hence the need for clever ways of finding it, and for estimating the size of what remains indirectly.

Check with math

① Test driving leaves rare hazards behind

As in ② of 5.1, the probability of never meeting a hazardous situation that occurs at rate λ (per km) in d km of driving is $e^{-\lambda d}$. When there are many kinds of hazardous situation, the expected number of kinds that remain unfound is, by adding expected values as in 1.4,

$$E[\text{kinds remaining unfound}] = \sum_i e^{-\lambda_i d}$$

Suppose there are 10 “common” kinds met once per 100,000 km and 10 “rare” kinds met only once per 10 million km (illustrative numbers):

Distance driven	Common kinds remaining (of 10)	Rare kinds remaining (of 10)
300,000 km	about 0.5	about 9.7
30 million km	almost 0	about 0.5

After 300,000 km, almost all the common kinds have been found, but almost all the rare ones remain. Finding nearly all the rare kinds takes 100 times the distance. What is more, as seen in 1.9, reality is often “heavy-tailed”: the rarer the situations, the more kinds of them there are. Shrinking area 3 through test driving alone is hard, and methods that actively search for hazards are needed (5.6–5.8).

② Estimating the share of kinds not yet seen

What is unknown cannot be counted, but there is a way to estimate “the probability that the next situation met is of a kind never seen before”. If, among N recorded events (the causes of driver interventions or near misses, for example), $N_1$ kinds appeared exactly once, then

$$P(\text{next is a new kind}) \approx \frac{N_1}{N}$$

This is called the Good–Turing estimate. The idea is that “many kinds seen only once” is a sign that “many kinds have not been seen at all”. With 30 once-only kinds in 1,000 records, it is 3%. If further driving brings the records to 10,000 and the once-only kinds fall to 12, it is 0.12%. A falling rate of meeting new kinds is one piece of evidence that area 3 is shrinking.

This estimate assumes, however, that events will keep occurring under the same conditions (the same ODD, similar driving). Kinds of places or weather never passed through in testing do not appear in the records at all. The result also depends on how kinds are defined (what counts as the same kind).

③ Estimating residual risk including the unknown

The danger that finally remains (the residual risk) is the sum of the known hazards left after measures (what remains of area 2) and an estimate for the unknown hazards (area 3):

$$R_{\text{residual}} = R_{\text{known}} + R_{\text{unknown}}$$

For example, suppose the known hazards after measures add up to $2 \times 10^{-8}$ per km. If recorded events occur once per 1,000 km, 0.12% of them (②) are new kinds, and a new kind is assumed to lead to harm with probability 1% when met, then

$$R_{\text{unknown}} \approx 10^{-3} \times 0.0012 \times 0.01 = 1.2 \times 10^{-8}\ \text{per km}$$

for a total of $3.2 \times 10^{-8}$ per km (illustrative numbers). The unknown part can be as large as the known part. This total is then compared with the acceptance criteria (5.5). Because the estimate for the unknown depends heavily on its assumptions, a conservative (larger) value is used, and watching continues after release (5.11).

Summary

SOTIF divides situations into four areas by “known or unknown” and “safe or hazardous”, finding unknown hazards to make them known (area 3 → 2) and fixing them to make them safe (area 2 → 1)
Test driving finds common hazards but leaves almost all the rare ones. The rarer the situations, the more kinds there often are, so methods that actively search for hazards are needed
The share of kinds not yet seen can be estimated from the share of kinds seen exactly once, N₁/N (Good–Turing). Residual risk is judged by including an estimate for the unknown as well as what remains of the known

Quick quiz

Q1. Which area is each in? (a) Testing has shown that pedestrians are missed in strong low sun, but no measures have been taken yet. (b) The vehicle brakes hard for a 3D-looking advertisement painted on the road, something nobody has tested. (c) Driving on a motorway on a clear day, thoroughly tested with no problems.

(a) Area 2 (known hazardous), (b) area 3 (unknown hazardous), (c) area 1 (known safe). (b) moves to area 2 as soon as it is found.

Q2. Using the formula in ①, if there are 10 kinds each met once per million km, how many are expected to remain unfound after a million km?

10 × e^(−1) ≈ 10 × 0.37 = 3.7 kinds.

Q3. Among 500 records, 25 kinds appeared exactly once. What is the estimated probability that the next event is of a new kind?

25 ÷ 500 = 0.05, about 5%.

Q4. “A million kilometres of test driving found no new hazards, so there are no unknown hazards left.” What is wrong with this view?

Not finding any only means none were met in that ODD, with that kind of driving, over that distance. As ① shows, rare hazards mostly remain after such a distance. The size of what remains has to be estimated together with how much testing was done, and other methods used to search as well.