Part 5 · SOTIF & Validation

5.10 Safety cases

Building the “why it is safe” argument with GSN

10-minute readPrerequisites: 5.5 Acceptance criteria and validation targets, 4.7 Three pillars of safety▶ Video (about 4 min)

What you will learn

Explain what a safety case shows and its elements (claims, argument strategies, evidence, context)
Read a GSN diagram that breaks a claim down into smaller claims and evidence
Confirm with simple calculations the strength of arguments supported by different kinds of evidence, the effect of the weakest claim, and the idea of eliminating defeaters

Build intuition

Part 5 has covered many activities so far: hazard analysis, triggering conditions, acceptance criteria, scenario testing, simulation, and how to check AI components. What is needed at the end is to connect their results and show, in a form a third party can follow, that “therefore this system can be said to be safe enough under these conditions”. That is the safety case. A safety case is made up of these elements:

Claims (goals): what is to be shown, such as “… is safe enough”
Argument strategies: how a claim is broken down to be shown (for example, by cause of hazard)
Evidence: test results, analysis records, dataset records and so on
Context: the conditions under which the claim holds (the ODD, how the system is used and so on)

A widely used way to draw this is GSN (Goal Structuring Notation). A claim at the top is broken down into smaller claims below it, with evidence supporting the bottom.

Part of a safety case written in GSN (schematic). Rectangles are claims, parallelograms are strategies, circles are evidence, rounded rectangles are context, and diamonds mark parts not developed in this figure.

In automated driving there is also the standard UL 4600, which sets out what a safety case for an autonomous product should contain, and which also calls for the argument to be checked continually against data gathered after release.

Check with math

① Two-legged arguments are strong — if independent

A single claim is sometimes supported by two different kinds of evidence. For example, “hazards from insufficiencies are small enough” might be shown separately by (a) scenario test results and (b) an analysis of triggering conditions. If each piece of evidence has a 10% chance of being misleading (making the claim look true when it is not), then, if the two are independent, the probability that both are misleading is

$$0.1 \times 0.1 = 0.01$$

— much smaller than with one leg (10%). But if the two pieces of evidence rely on the same simulation model or the same data, they are not independent. If a shared assumption is wrong with probability 5%, and both legs are then misleading, the probability is about 0.05 + 0.95 × 0.01 ≈ 0.06 — almost entirely decided by the shared assumption. As with the common causes of 1.1, the legs of an argument should be different kinds of evidence as far as possible.

② A chain of claims is as strong as its weakest link

When a claim above holds only if all the claims below hold (the three claims in the figure), the overall confidence is the product of the confidences in the claims below:

$$P(\text{top claim}) = P(\text{faults}) \times P(\text{insufficiencies}) \times P(\text{attacks})$$

Estimating them at 0.99, 0.95 and 0.98 gives 0.99 × 0.95 × 0.98 ≈ 0.92. Strengthening the insufficiency claim to 0.99 raises this to 0.96. As with errors in series in 4.2, strengthening the evidence for the weakest claim is the shortcut to strengthening the whole. The numbers are for illustration: in practice confidence is hard to express as a number, and giving reasons in words is at the centre.

③ Eliminating defeaters

An effective way to strengthen an argument is to list defeaters — “in what cases would this claim fall?” — and eliminate them one by one. For the claim “scenario testing is sufficient”, possible defeaters include “the simulation may differ from reality” (5.7), “the list of scenarios may have gaps” (5.6) and “the estimate of the real-world distribution may be off” (5.8). If three defeaters remain unresolved, each with a 10% chance of being right (independently), the probability that none of them is right is

$$0.9^3 \approx 0.73$$

Adding evidence to eliminate two of them raises this to 0.9. Defeaters that cannot be eliminated are written down openly rather than hidden, and compensated for with ODD restrictions and monitoring in the field.

Summary

A safety case builds the claim “this system is safe enough under these conditions” from argument strategies and evidence. GSN draws the claims, strategies, evidence and context
Supporting a claim with two different kinds of evidence strengthens the argument, but a shared assumption then decides its strength. When all the claims below are needed, the weakest one decides the whole
List and eliminate defeaters; write down those that cannot be eliminated and compensate with the ODD and monitoring. Check the argument with independent eyes and keep updating it with changes and field data

Quick quiz

Q1. In GSN, is “the record of the residual risk estimate” a claim, a strategy, evidence or context?

Evidence (drawn as a circle).

Q2. In an argument supported by two pieces of evidence, each with a 20% chance of being misleading, what is the probability that both are misleading if they are independent?

0.2 × 0.2 = 0.04, or 4%.

Q3. In ②, if the three claims have confidences of 0.98, 0.90 and 0.99, what is the confidence in the top claim?

0.98 × 0.90 × 0.99 ≈ 0.87. Strengthening the weakest claim, at 0.90, first is most effective.

Q4. “We have gathered all the test results, so the safety case is complete.” What is wrong with this view?

Collecting evidence is not enough: it must be shown which claim each piece of evidence supports and how (the argument), and under what context the claims hold. Whether the defeaters have been eliminated must also be checked, and the case must keep being updated with changes and field data.