Part 1 · Probability & Statistics

1.8 Statistics of rare events

How far must you drive without an accident?

8-minute readPrerequisite: 1.3 Probability distributions (Poisson distribution)▶ Video (about 5 min)

What you will learn

Explain what “zero accidents” shows — and what it does not show
Quickly estimate the required accident-free distance with the rule of three
Explain why driving tests alone struggle to demonstrate safety, and the direction SOTIF takes to address this

Build intuition

An automated vehicle has driven a total of 100,000 km on test tracks and public roads without a single accident. Can we say that its accident rate is zero?

No. Even if the car’s true performance were “one accident per 100,000 km”, the probability of finishing 100,000 km with zero accidents purely by chance is about 37%. More than one time in three, you would see “zero accidents”.

So how far must we drive to say it is “at least safer than one accident per 100,000 km”? The farther we drive, the less likely this coincidence becomes: “the true rate is one per 100,000 km, yet there were still no accidents”.

Even a car whose true accident rate is exactly at the target ends with zero accidents 37% of the time after driving 1× the target distance. At 3×, that probability drops to 5%.

If there are still zero accidents after driving three times the target distance, we can say: “If the true performance were at the target (or worse), a result like this would occur with a probability of 5% or less.” This is the condition for claiming, with 95% confidence, that the vehicle is safer than the target.

The rule of three
If the target is “no more than one accident per X km”, then driving 3X km without an accident lets you say, with 95% confidence, that the target is met.

Check with math

The number of events that occur “at a constant rate, randomly and independently of each other” — like accidents — can be described by the Poisson distribution (1.3). Let $\lambda$ (lambda) be the accident rate per km, $d$ the distance driven, and $N$ the number of accidents. The average number of accidents is $\lambda d$, and the probability of zero accidents is:

$$P(N = 0) = e^{-\lambda d}$$

$e \approx 2.718$ is a constant known as Euler’s number. Here you only need to remember two values: $e^{-1} \approx 0.37$ and $e^{-3} \approx 0.05$ (the 37% and 5% in the graph).

Assume the true accident rate is exactly the target value $\lambda_0$, and find the distance at which the probability of “zero accidents anyway” falls to 5% or less.

$$e^{-\lambda_0 d} \le 0.05 \quad\Longrightarrow\quad d \ge \frac{-\ln 0.05}{\lambda_0} \approx \frac{3.0}{\lambda_0}$$

Since $1/\lambda_0$ is the “distance per accident” X, we get $d \approx 3X$. That is where the rule of three comes from. Changing the confidence level changes the factor: about 2.3 for 90% and about 4.6 for 99%.

Target (distance per accident)	Required accident-free distance (95% confidence)
1 per 100,000 km	about 300,000 km
1 per 1 million km	about 3 million km
1 per 100 million km	about 300 million km
U.S. fatality rate 1.09 per 100 million miles ≈ 1 per 150 million km	about 440 million km (about 275 million miles — roughly 11,000 times around the Earth)

Try it yourself

Advanced: what if an accident happens along the way?

Even if accidents occur during testing, you may still be able to say the target is met. But the required distance grows, and the factor “3” increases as follows (95% confidence).

Observed accidents k	0	1	2	3
Required distance (multiple of target distance X)	3.0	4.7	6.3	7.8

In general, the distance is calculated with the chi-square distribution using the formula below. When k = 0, $\chi^2_{0.95}(2)/2 \approx 3.0$, which matches the rule of three.

$$d \ge \frac{\chi^2_{0.95}(2k+2)}{2\lambda_0}$$

ISO 21448, the SOTIF standard, also gives examples of estimating the driving distance (or time) needed for validation using similar statistical reasoning. We cover this in detail in 5.5.

Advanced: updating prior evidence with mileage

The rule of three assumes that nothing is known before driving starts. With Bayes’ theorem from 1.2, evidence held before driving (analysis, simulation, the record of a previous system) can be entered as a prior belief and updated with the mileage. The simplest form expresses the prior evidence as “worth $d_0$ km of driving with one accident”. After $d$ accident-free km, the upper bound on the accident rate (95% confidence) is then

$$\lambda \le \frac{3}{d_0 + d}$$

With $d_0 = 0$ (no prior evidence) this is the rule of three itself. For a target of “no more than one per 100 million km”, the rule of three needs 300 million accident-free km, but if the prior evidence can be said to be worth “200 million km with one accident”, 100 million km of real driving is enough. If an accident happens along the way, the factor grows to 4.7, 6.3, … as in the table above.

The important point is that a generous estimate of $d_0$ can shrink the required distance without limit. How many kilometres the prior evidence is really worth must be shown in a form a third party can follow (→ 5.10). How far simulation results can stand in for real driving (5.7), and how much of the earlier record carries over after a software update (see “Connection” below), are questions about this same $d_0$.

Summary

Zero accidents does not prove a zero accident rate. After driving only 1× the target distance, zero accidents happens by chance 37% of the time
Rule of three: to show “no more than one accident per X km” with 95% confidence, drive about 3X km without an accident
Demonstrating human-level safety through real-world driving alone takes hundreds of millions of kilometres, so SOTIF combines scenarios and simulation

Quick quiz

Q1. To show with 95% confidence that the target “no more than 1 accident per 1 million km” is met, how far must you drive without an accident?

About 3 million km — 3 × 1 million km, by the rule of three.

Q2. A car drove 500,000 km with zero accidents. What upper bound on its accident rate can you state with 95% confidence?

3 ÷ 500,000 km = 6 accidents per 1 million km. In other words, you can only say it is “no worse than about one accident per 170,000 km”. Note that you cannot claim “no more than one accident per 500,000 km”.

Q3. While testing against a target of “no more than 1 accident per 100,000 km”, one accident occurred. How far must you drive in total, with the count staying at one, to claim the target is met with 95% confidence?

About 470,000 km. With one accident the factor is about 4.7, so 4.7 × 100,000 km. With zero accidents, 300,000 km would have been enough — so a single accident increases the required distance by nearly 1.6 times.