Appendix

Putting a number on residual risk

How the estimate is built, and where each piece comes from

To build the claim “the residual risk of this system is below the acceptance criterion” out of numbers, the statistics of Part 1 and the activities of Part 5 have to be joined into a single chain. This page lays that chain out on one sheet. All the numbers are illustrative; the point is to show which factor is covered in which lesson.

The claim Known hazards Unknown hazards Taxonomy validity Bias Combining Worked example Lesson map

1. Decide the form of the claim first

What is to be shown is that the rate of harm per km within the ODD, $R$, is no more than the acceptance criterion $R_{\text{acc}}$ (5.5). When this is shown in numbers, it comes with a confidence level (for example, as a 95% one-sided upper bound) and its context (the ODD, the use, the software version). Following the areas of 5.2, $R$ is split in two.

$$R = R_{\text{known}} + R_{\text{unknown}} \le R_{\text{acc}}$$

$R_{\text{known}}$ is the risk that remains from hazards that have been found and mitigated; $R_{\text{unknown}}$ is the risk from hazards not yet found.

Decomposition of the residual-risk estimate. The top claim splits into a known and an unknown part, and each into three factors. Each factor has its own source and its own way of getting a range.

2. Known hazards: break down by condition

For each triggering condition found (5.3), multiply “the rate of meeting the condition”, “the probability of hazardous behaviour when it is met” and “the probability that the hazardous behaviour leads to harm”, and add up (the exposure weighting of 1.4; “rates add” from 1.3).

$$R_{\text{known}} = \sum_i w_i \, p_i \, c_i$$

$w_i$: the rate of meeting condition $i$ (per km). Measured from naturalistic driving data and field data (5.6, 5.11)
$p_i$: the probability of hazardous behaviour under condition $i$. Measured by targeted tests and simulation (5.6–5.9). With zero failures, the rule of three gives the upper bound $3/n$ (1.8)
$c_i$: the probability that the hazardous behaviour leads to harm. From the analysis of controllability and severity and the effect of fallbacks (5.4)

The three factors have different sources, so each is estimated from its own data and given its own range (1.6).

3. Unknown hazards: Good–Turing is only the entrance

What the Good–Turing estimate (5.2) tells you is the share of encounters whose kind has never been seen before, $N_1/N$ (where $N_1$ is the number of kinds that appeared exactly once among the $N$ recorded events). That is not a rate of hazard. To turn it into a rate of harm, two more factors are multiplied in.

$$R_{\text{unknown}} \approx \underbrace{\frac{N_1}{N} \times s}_{\text{new-kind rate}} \times q \times c$$

$s$: the number of scenes met per km. The new-kind rate is then “the rate of meeting a new kind (per km)”
$q$: the share of new kinds that are unsafe (that trigger a functional insufficiency). Estimated from the share of kinds found so far that turned out to be hazardous. The counts are small, so use the upper end of a confidence interval (1.6), or the rule of three when the count is zero (1.8)
$c$: the probability that an unsafe scene leads to harm. The same controllability analysis as for the known part (5.4)

$N_1/N$ falls as mileage grows, and $c$ can be lowered by fallbacks. Holding the unknown part as a number means that “how many more km to drive” and “which countermeasure to add” are decided by numbers (see the worked example below).

4. Validity of the scenario taxonomy

$N_1/N$ depends entirely on how “kinds” are cut. Too coarse, and everything has “been seen before”: $N_1$ comes out small and the unknown is underestimated. Too fine, and $N_1$ balloons and it is overestimated. Three conditions make a taxonomy defensible.

It corresponds to failure mechanisms: check that the failure rate is roughly uniform within a kind, using the scatter inside each kind. Split kinds with large scatter (5.3)
Sensitivity to granularity: vary the cut across the functional, logical and concrete levels (5.6), show how $N_1/N$ moves, and use the conservative value
Cross-check with an independent taxonomy: confirm that a taxonomy built by different people or from a different viewpoint does not give a very different estimate

This is a matter of argument rather than statistics. It goes into the safety case (5.10) as the “grounds for the taxonomy”, in a form that answers the defeater “the taxonomy may be too coarse”.

5. Sampling bias

$N$, $N_1$ and $w_i$ are set by where you drove. On sunny motorways alone, new kinds that appear on rainy nights are never found, and the $w_i$ are off too. Confidence intervals only reflect random scatter, so this bias lies outside them (1.6). The remedies are:

Estimate exposure and failure rates separately, from separate data: $w_i$ from naturalistic driving, $p_i$ from targeted tests. Mixing them breaks the “same conditions” assumption (1.5)
Stratify by ODD condition: estimate within each stratum and weight by exposure (1.4). Map deliberately targeted searches back to the real distribution with importance-sampling weights (5.8)
Watch the bias in field data: only what is detected gets recorded. Decide what to record before release (5.11)

6. Combine and compare with confidence

The simplest method is to multiply and add the upper bounds (one-sided 95%) of each factor. It is conservative (the true confidence is higher than 95%) but easy to explain. For more accuracy, draw random values from each factor’s distribution and compute $R$ thousands of times (Monte Carlo, 1.5), or resample the data (the bootstrap, 1.9), and take the 95% point of $R$.

The range for $R_{\text{unknown}}$, however, is a model-dependent value that rests on the taxonomy and the assumption for $q$; its meaning differs from a frequentist “95%”. That is why, in the safety case, this estimate is not presented as a stand-alone claim but together with its context, the grounds for the taxonomy, the sensitivity analysis and the defeaters (5.10), and is kept updated with field data (5.11).

7. Worked example (illustrative)

Let the acceptance criterion be $R_{\text{acc}} = 10^{-8}$ per km (one per 100 million km). For the known part, suppose three triggering conditions have been found:

Triggering condition	Exposure $w_i$ (/km)	Upper bound on $p_i$	P(harm) $c_i$	Product (/km)
Strong glare	0.02	$3 \times 10^{-5}$ (0 failures in 100,000)	0.002	$1.2 \times 10^{-9}$
Heavy rain	0.01	$1 \times 10^{-4}$ (0 failures in 30,000)	0.002	$2.0 \times 10^{-9}$
Close cut-in	0.005	$1 \times 10^{-4}$ (0 failures in 30,000)	0.005	$2.5 \times 10^{-9}$
Total $R_{\text{known}}$				$5.7 \times 10^{-9}$

For the unknown part, suppose a million km were driven and a million scenes recorded ($s = 1$ per km), $N_1 = 500$ kinds appeared exactly once, 10 of the 2,000 kinds found so far turned out to be unsafe (95% upper bound on $q$ ≈ 0.8%), and the probability that an unsafe scene leads to harm is $c = 2\%$.

$$R_{\text{unknown}} \approx \frac{500}{10^6} \times 1 \times 0.008 \times 0.02 = 8 \times 10^{-8}$$

The total is about $8.6 \times 10^{-8}$, more than eight times the criterion. The unknown part dominates: no amount of extra testing of the known hazards would be enough. What works is the factors of the unknown part. If mileage grows to 10 million km and the new-kind rate falls to $10^{-4}$ per km, and an improved fallback brings $c$ down to 0.2%, then

$$R_{\text{unknown}} \approx 10^{-4} \times 0.008 \times 0.002 = 1.6 \times 10^{-9}$$

and the total, $7.3 \times 10^{-9}$, is below the criterion. The margin is only about 30%, though, so the sensitivity analysis of the taxonomy and monitoring in the field are indispensable. Holding the unknown part as a number is what makes “what to do, and how much, to reach the criterion” a matter of calculation.

8. Map from factors to lessons

Factor	Source	Range and cautions	Lessons
Acceptance criterion $R_{\text{acc}}$	Comparison with humans, ALARP, positive risk balance	State the confidence level and one-/two-sided	5.5, 1.6
Exposure $w_i$	Naturalistic driving data, field data	Stratify; confidence interval	1.4, 5.6, 5.11
Conditional failure rate $p_i$	Targeted tests, simulation, evaluation of AI components	$3/n$ with zero failures; simulation credibility; updating prior evidence	1.8, 5.7, 5.9
P(harm) $c$	Controllability and severity analysis, fallbacks	Conservative values; extrapolation from near misses	5.4, 1.9
New-kind rate $N_1/N$	Taxonomy of recorded scenes	Taxonomy validity; sensitivity to granularity	5.2, 5.6
Unsafe share $q$	Record of kinds found	Small counts; use the upper bound	1.6, 1.8
Combining	Products and sums of bounds, Monte Carlo, bootstrap	The unknown part is model-dependent	1.5, 1.9
Context and argument	ODD, grounds for the taxonomy, defeaters	Keep as a living document	5.10, 5.11

Examples of the context to write into the safety case: the definition of the ODD, the scene taxonomy and its granularity, the definition of $s$, the grounds for $q$ and $c$, the credibility of tests and simulation, the data period and software version, and the indicators watched in the field with the conditions for updating.

Triggering condition	Exposure \(w_i\) (/km)	Upper bound on \(p_i\)	P(harm) \(c_i\)	Product (/km)
Strong glare	0.02	\(3 \times 10^{-5}\) (0 failures in 100,000)	0.002	\(1.2 \times 10^{-9}\)
Heavy rain	0.01	\(1 \times 10^{-4}\) (0 failures in 30,000)	0.002	\(2.0 \times 10^{-9}\)
Close cut-in	0.005	\(1 \times 10^{-4}\) (0 failures in 30,000)	0.005	\(2.5 \times 10^{-9}\)
Total \(R_{\text{known}}\)				\(5.7 \times 10^{-9}\)

Factor	Source	Range and cautions	Lessons
Acceptance criterion \(R_{\text{acc}}\)	Comparison with humans, ALARP, positive risk balance	State the confidence level and one-/two-sided	5.5, 1.6
Exposure \(w_i\)	Naturalistic driving data, field data	Stratify; confidence interval	1.4, 5.6, 5.11
Conditional failure rate \(p_i\)	Targeted tests, simulation, evaluation of AI components	\(3/n\) with zero failures; simulation credibility; updating prior evidence	1.8, 5.7, 5.9
P(harm) \(c\)	Controllability and severity analysis, fallbacks	Conservative values; extrapolation from near misses	5.4, 1.9
New-kind rate \(N_1/N\)	Taxonomy of recorded scenes	Taxonomy validity; sensitivity to granularity	5.2, 5.6
Unsafe share \(q\)	Record of kinds found	Small counts; use the upper bound	1.6, 1.8
Combining	Products and sums of bounds, Monte Carlo, bootstrap	The unknown part is model-dependent	1.5, 1.9
Context and argument	ODD, grounds for the taxonomy, defeaters	Keep as a living document	5.10, 5.11