Part 5 · SOTIF & Validation

5.6 Scenario-based testing

ODD, scenarios and parameter spaces

10-minute readUpdated 2026-10-13Prerequisites: 5.5 Acceptance criteria and validation targets, 4.5 Prediction and motion planning (the following-distance formula)▶ Video coming soon

What you will learn

Explain, with examples, the idea of scenario-based testing and the three levels of scenario: functional, logical and concrete
Calculate how combinations in a parameter space explode, and how physics can locate the boundary so that tests can be concentrated there
Calculate what the rule of three lets you say from scenarios sampled at random from real-world distributions
(Advanced) Explain how orthogonal arrays and pairwise testing cut down the combinations when not everything can be tested

Build intuition

As we saw in 5.5, showing safety through driving distance alone is all but impossible. Scenario-based testing instead systematically selects situations likely to lead to danger (scenarios) and checks them while varying the conditions. Rather than waiting to meet them by chance, it deliberately constructs the situations to be checked.

Scenarios are written at three levels of abstraction:

Level	How it is written	Example
Functional scenario	An outline of the situation, in words	On a motorway, a vehicle in the next lane cuts in ahead of the ego vehicle
Logical scenario	The parameters describing the situation and their ranges	Ego speed 60–120 km/h, distance at cut-in 5–60 m, closing speed 0–20 m/s, lateral speed 0.5–2 m/s
Concrete scenario	Specific values for the parameters, making a single test	Ego speed 100 km/h, distance 25 m, closing speed 10 m/s, lateral speed 1.2 m/s

The parameter ranges of a logical scenario are set from the ODD (4.1) and from real traffic data. Sources of scenarios include accident records, naturalistic driving data recorded from real driving, hazard analysis (5.4) and analysis of triggering conditions (5.3), and test scenarios specified by regulations.

Parameter space of a cut-in logical scenario (schematic). Each point is a concrete scenario. Results far from the boundary are practically known in advance, so tests are concentrated near the boundary (small points).

The figure shows the parameter space formed by two parameters of the cut-in logical scenario (distance and closing speed). The closer and faster the cut-in, the more dangerous, and a boundary between collision and no collision emerges. Results far from the boundary are practically known in advance, so concentrating tests near the boundary is efficient.

Check with math

① Combinations explode quickly

If a logical scenario has k parameters and each is divided into n levels and every combination is tested, the number of concrete scenarios is

$$N = n^k$$

Four parameters at ten levels give 10⁴ = 10,000 combinations; six give 10⁶ = one million. At one minute per simulation, a million takes about two years. Adding weather, lighting, the number of surrounding vehicles and so on increases it further, so testing everything on a grid is impossible. Techniques are needed: sampling at random, concentrating near the boundary, and focusing on the dangerous regions (5.8).

② Estimating the boundary with physics

If the cutting-in vehicle keeps a constant speed, the minimum distance at which the ego vehicle, braking with deceleration a after a response time $t_r$, can avoid a collision is, by the same reasoning as the following distance in 4.5,

$$d^* = \Delta v\,t_r + \frac{\Delta v^2}{2a}$$

(Δv is the closing speed). With $t_r = 0.5$ s and $a = 4\ \mathrm{m/s^2}$, Δv = 10 m/s gives d* = 5 + 12.5 = 17.5 m. The boundary line in the figure is drawn with this formula. Estimates like this indicate where tests should be concentrated. In a real system, though, perception delays and errors, and the way the cutting-in vehicle moves, shift the boundary. Finding that shift is what the testing is for.

③ Grid results do not directly give the size of the danger

Of the 30 grid points in the figure, 11 end in a collision. That does not mean “37% of cut-ins end in a collision”, because the grid does not reflect how often each situation actually occurs. To estimate real-world danger, concrete scenarios are sampled at random according to the distribution of the parameters, obtained from naturalistic driving data and the like. If N scenarios selected this way produce zero collisions, the rule of three from 1.8 says that, with 95% confidence, the probability of a collision per cut-in is

$$p \le \frac{3}{N}$$

With N = 3,000, that is at most 0.1%. To show a collision probability per cut-in of 10⁻⁶ or less, as in the table in ③ of 5.5, N = 3 million is needed. That is hardly feasible except in simulation (5.7), and methods that raise efficiency by focusing on the dangerous regions are needed on top (5.8).

Advanced: cutting down the combinations to test (design of experiments and pairwise testing)

As ① showed, not every combination can be tested, so the combinations to test have to be chosen deliberately. The branch of statistics for this is the design of experiments. The parameters under investigation are called factors and their steps are called levels. When the effect of one condition depends on the level of another, that is an interaction; the “85% on a rainy night” of 5.3 is an example.

A standard tool is the orthogonal array. Take four parameters — ego speed, distance at the cut-in, closing speed and lateral speed — each at three levels, small, medium and large (1, 2, 3). All combinations number 3⁴ = 81, but the L9 orthogonal array below needs only 9 tests.

Test	Ego speed	Distance	Closing speed	Lateral speed
1	1	1	1	1
2	1	2	2	2
3	1	3	3	3
4	2	1	2	3
5	2	2	3	1
6	2	3	1	2
7	3	1	3	2
8	3	2	1	3
9	3	3	2	1

In this table every level appears three times in each column, and any two columns contain each of the 9 pairs of levels exactly once. This lets the effect of each parameter be estimated separately, and every pair of conditions is tried at least once.

The pairwise testing of 5.3 asks only that every pair appears “at least once”, a looser condition than an orthogonal array. As a result, a short list of tests can be generated by a tool even when the number of conditions or levels does not fit the shape of an orthogonal array.

There are four things to keep in mind:

Continuous boundaries are not found: a table of combinations only tells you whether you passed “near” the boundary (the curve in the figure). The boundary is pinned down with the estimate in ② and the counterexample search of 5.8.
Problems from three or more factors are missed: to try every triple, 6 conditions at 3 levels give 20 × 3³ = 540 triples; each test covers 20, so at least 27 tests are needed.
Combinations outside the ODD are removed: combinations that cannot occur within the ODD are excluded when planning.
How likely they are stays unknown: what you learn is whether a problem occurs in that combination, not how likely it is in the real world. The probability is estimated by random sampling from the real-world distribution, as in ③.

The flow is: find candidates with orthogonal arrays or pairwise testing → search near the boundary (②, 5.8) → estimate the probability (③, and the importance sampling of 5.8).

Summary

Scenario-based testing systematically selects situations likely to lead to danger and checks them while varying the conditions. Scenarios are written at three levels — functional, logical and concrete — with parameter ranges set from the ODD and real traffic data
Parameter combinations grow as $n^k$, so not everything can be tested. Physics and similar estimates locate the boundary, and tests are concentrated near it
Grid results do not directly give the size of the danger. If N scenarios sampled at random from the real-world distribution produce zero collisions, p ≤ 3/N
(Advanced) The design of experiments — orthogonal arrays and pairwise testing — cuts down the combinations, but the shape of the boundary and how likely each case is must be found separately

Quick quiz

Q1. Which level is each scenario — functional, logical or concrete? (a) Being cut in on ahead on a motorway. (b) Ego 100 km/h, distance 25 m, closing speed 10 m/s. (c) Ego 60–120 km/h, distance 5–60 m, closing speed 0–20 m/s.

(a) Functional; (b) concrete; (c) logical.

Q2. With five parameters, each divided into eight levels, how many concrete scenarios are there if every combination is tested?

8⁵ = 32,768.

Q3. Using the formula in ② ($t_r = 0.5$ s, $a = 4\ \mathrm{m/s^2}$), what is the minimum distance that avoids a collision at a closing speed of 8 m/s? What if the cut-in happens at 10 m?

d* = 8 × 0.5 + 64 ÷ 8 = 4 + 8 = 12 m. At 10 m the cut-in is closer than that, so by this formula it falls on the collision side.

Q4. In 30,000 concrete scenarios sampled at random from the real-world distribution, there were no collisions. What can be said about the probability of a collision per cut-in?

3 ÷ 30,000 = 10⁻⁴: with 95% confidence, at most 0.01%.

Q5. (Advanced) Six conditions each have three levels. What is the minimum number of tests needed to try every combination of levels of any three conditions at least once?

There are 20 × 3³ = 540 triples of levels, and one test covers 20 of them, so 540 ÷ 20 = 27: at least 27 tests.