Part 5 · SOTIF & Validation

5.3 Functional insufficiencies and triggering conditions

Sun glare, heavy rain, unfamiliar objects

10-minute readPrerequisites: 5.1 What is SOTIF?, 1.6 Estimation and confidence intervals▶ Video (about 5 min)

What you will learn

Explain, with examples, the two kinds of functional insufficiency (of specification and of performance) and the triggering conditions that bring them out
Explain how to search by crossing categories of conditions with the parts of the system, and how to check detection rates per condition with confidence intervals
Confirm with simple calculations how triggering conditions can appear in combination (pairwise testing) and how performance boundaries relate to the ODD

Build intuition

As we saw in 5.1, SOTIF calls the source of dangers that arise without any fault a functional insufficiency, and the situation that brings it out a triggering condition. Functional insufficiencies come in two kinds:

Kind	Meaning	Examples
Insufficiency of specification	The function’s specification (what it handles, and how) leaves some situations unconsidered	Hand signals from a flagger at roadworks are not covered by the specification; lanes are assumed always to be marked with white lines
Insufficiency of performance	Even when built to specification, the performance of sensing, perception or actuation falls short	A camera misses a pedestrian in low sun; fog shortens LiDAR’s range; stopping distances lengthen on wet roads

To search for triggering conditions systematically, a set of categories is commonly used:

Category	Example triggering conditions	Insufficiencies likely to surface
Environment (weather, light)	Heavy rain, fog, snow, low sun, the light change at tunnel entrances and exits	Reduced camera and LiDAR performance
Road and surroundings	Faded lane markings, roadworks, sharp bends, wet roads shining like mirrors	Lane perception, localization
Other road users	Sudden cut-ins, rule-breaking movements, people stepping out from behind obstructions	Specification and performance of prediction and planning
Objects	Unusually shaped vehicles, debris, animals, pictures painted on the road	Object detection (absent from training data)
Own vehicle’s condition	Dirt or snow on sensors, misaligned sensors, heavy loads	All sensors, actuation
People (driver and others)	Over-trust in driver assistance, overlooked displays	Misuse, specification of displays and warnings

The basic method is to build a table crossing these categories with the parts of the system (sensing, perception, prediction, planning, actuation) and to ask, one cell at a time, “how does this part weaken under this condition?”. Information on past accidents and recalls, expert knowledge and analysis of driving data are added to this. Candidates found this way are then tested to confirm whether they really lead to hazardous behaviour.

Example of pedestrian detection rate against rain intensity (illustrative numbers). The point where a sensor starts to fall below the requirement marks the boundary of a triggering condition for that sensor.

The camera in the figure drops below the requirement (95%) at 7.5 mm/h, inside the ODD limit of 10 mm/h used as an example in 4.1. This means “there is an insufficiency of performance inside the ODD”, and measures are needed.

Check with math

① Check condition by condition

Triggering conditions are found by measuring performance per condition. Here is an example of testing the pedestrian detection rate split by rain intensity. The intervals use $\hat{p} \pm 2\sqrt{\hat{p}(1-\hat{p})/n}$ from 1.6.

Condition	Detections / trials	Detection rate	95% confidence interval
No rain	1,960 / 2,000	98.0%	97.4–98.6%
Rain 5–10 mm/h	380 / 400	95.0%	92.8–97.2%
Rain 10 mm/h or more	180 / 200	90.0%	85.8–94.2%
Overall	2,520 / 2,600	96.9%	—

Looking only at the overall 96.9%, the 95% requirement seems to be met. But in rain of 10 mm/h or more, even the upper end of the interval (94.2%) is below the requirement: clearly a triggering condition. For 5–10 mm/h the interval straddles 95%, so more testing is needed to decide. Averages dilute weak conditions (2.8), so looking condition by condition is the basic rule.

② Triggering conditions that appear in combination

Conditions that cause no trouble one at a time can cause trouble together. For example, detection might be 96% at night and 96% in rain — both meeting the requirement — yet drop to 85% on rainy nights (headlights reflecting off the wet road, for instance). Not missing such combinations requires testing conditions in combination. With six conditions of three levels each, all combinations number

$$3^6 = 729$$

Road-testing them all is a lot of work, but “trying every combination of two conditions at least once” takes far fewer. The pairs of levels of two conditions number $\binom{6}{2} \times 3^2 = 15 \times 9 = 135$, and one test covers 15 such pairs at once, so at least 135 ÷ 15 = 9 tests are needed; with a well-arranged plan, a dozen or so tests cover every pair at least once. This is called pairwise testing (combinatorial testing). It builds on the known fact that most software defects are triggered by combinations of only a few conditions.

③ Performance boundaries and the ODD

Approximating the camera’s detection rate in the figure as a function of rain intensity r (mm/h),

$$p(r) = 98 - 0.4\,r\ \ (\%)$$

It starts to fall below the required 95% where 98 − 0.4r = 95, that is, at r = 7.5 mm/h. Since this is below the ODD limit (10 mm/h), a measure must be chosen:

Narrow the ODD: lower the rain limit to 7.5 mm/h (4.1)
Improve performance: combine with LiDAR or radar (the fusion of 4.3)
Change the behaviour: slow down in heavy rain, leaving enough margin to stop even after a miss (4.5)

Which to choose is decided by balancing how widely the function can be used against the effort needed. LiDAR, for its part, drops below the requirement at about 18 mm/h, outside the ODD, so it causes no problem within this ODD.

Summary

Functional insufficiencies are of two kinds — of specification (situations not considered) and of performance (performance falls short) — and triggering conditions bring them out
Triggering conditions are searched for with tables of condition categories × parts of the system and by measuring performance split by condition. Averages dilute weak conditions, so check condition by condition, with confidence intervals
Some conditions appear only in combination (pairwise testing). If a performance boundary lies inside the ODD, respond by narrowing the ODD, improving performance or changing the behaviour

Quick quiz

Q1. Is each an insufficiency of specification or of performance? (a) A flagger at roadworks signals by hand, but the function’s specification does not cover that situation. (b) Fog shortens LiDAR’s range and the vehicle cannot stop in time.

(a) Specification; (b) performance.

Q2. In rain of 10 mm/h or more, a pedestrian was detected 282 times out of 300. What are the detection rate and 95% confidence interval? Can we say it is below the 95% requirement?

The rate is 94%, and the interval is 94% ± 2√(0.94 × 0.06 / 300) ≈ 94% ± 2.7%, about 91.3–96.7%. The interval straddles 95%, so more testing is needed before saying it falls short.

Q3. With five conditions of four levels each, how many combinations are there in total? How many tests at least are needed to try every pair of two conditions once?

All combinations: 4⁵ = 1,024. Pairs of levels of two conditions: $\binom{5}{2} \times 4^2 = 10 \times 16 = 160$; one test covers 10 pairs, so at least 16 tests are needed.

Q4. For the camera in ③ (p(r) = 98 − 0.4r), if the requirement were 96%, from what rain intensity would it fall short?

From 98 − 0.4r = 96, r = 5 mm/h. The stricter the requirement, the further the boundary of the triggering condition moves towards lighter rain.