Part 4 · How Automated Driving Works

4.6 Modular vs. end-to-end

Two architectures and what they mean for validation

10-minute readPrerequisites: 4.2 System overview, 4.5 Prediction and motion planning▶ Video (about 6 min)

What you will learn

Explain how the modular and end-to-end approaches are built, and the strengths and weaknesses of each
Confirm with simple calculations how small deviations compound in imitation learning, and the danger of evaluating by average error alone
Explain how validation changes with the architecture, and what to watch for when a rule-based monitor guards a learned component

Build intuition

In 4.2 we looked at the modular approach, which builds the system from separate parts: sensors, perception, prediction, planning and control. The end-to-end approach instead trains a single large neural network to go all the way from inputs such as cameras to the trajectory or controls. Large amounts of driving data, together with the advances in Transformers and foundation models covered in 3.5 and 3.6, have drawn much attention to it in recent years.

Modular vs. end-to-end (schematic). Learned models are also used inside the parts of a modular system, so the difference is not “whether learning is used” but “whether the system is built as separate parts”.

	Modular	End-to-end
How it is built	Part by part, combining learned parts with rule-based and optimization parts	Everything from input to driving is trained together on large amounts of driving data
Strengths	Each part can be tested, and the causes of errors are easier to pinpoint. Safety requirements are easy to allocate to parts	No information is discarded between parts. Can learn complex behaviour that rules cannot fully describe (such as negotiating with other road users). Tends to improve with more data
Weaknesses	Information that does not fit the hand-over format is lost (debris not in the object list, for example). Hand-written rules have gaps	Hard to see inside and to explain. Little data for rare situations. Parts cannot be tested separately
Main way of checking	Tests of each part + tests of the whole	Tests of the whole, situation by situation (simulation, scenarios)

In practice, systems do not fall neatly into one camp or the other. There are many in-between forms: systems split into parts but with learned models inside them, single models that are made to output intermediate results such as detections and predictions so they can be monitored, learned planners surrounded by rule-based monitors, and so on.

Check with math

① Small deviations compound (imitation learning)

Many end-to-end systems learn by imitation learning, copying human driving as a model to follow (there is also reinforcement learning, which learns by trial and error guided by rewards, used mainly in simulation). Imitation learning has a weakness. Human demonstrations almost always stay in the middle of the lane, so there are hardly any examples of “recovering from a slight deviation”. Once the model drifts even slightly, it enters states it has never seen, where its decisions cannot be trusted. One deviation invites the next.

Suppose that at each decision, made every 0.1 s, the probability of drifting into an unseen state is ε = 0.001 (once in 1,000). The probability of never drifting in 100 seconds (T = 1,000 decisions) is, by the same calculation as in 1.1 and 1.8,

$$P(\text{never drifts}) = (1-\varepsilon)^T \approx e^{-\varepsilon T} = e^{-1} \approx 0.37$$

So about 63% of 100-second drives enter an unfamiliar state at least once. If the model cannot recover after drifting, theory shows that the effect can grow as fast as the square of the driving time (Ross et al. in the references). Remedies include letting the model drive and re-teaching it the “right way back” from the states it drifts into (DAgger), and artificially creating “recovery” examples from camera images shifted to the left and right.

② Don’t be fooled by the average error

A common way to evaluate end-to-end systems is to measure, for each scene in recorded driving data, the gap between the trajectory the model outputs and the one the human actually drove. Because the model’s decisions do not affect the next scene, this is called open-loop evaluation. Looking only at the average here is dangerous. Suppose 95% of the evaluation data is simply driving straight at constant speed, and the remaining 5% is hard situations such as merging or sudden braking. A model that always answers “carry straight on at the current speed”, with errors of 0.2 m in easy scenes and 4 m in hard ones, has an average error of

$$0.95 \times 0.2 + 0.05 \times 4 = 0.19 + 0.20 = 0.39\ \text{m}$$

which looks small. Yet in the hard scenes it is of no use at all. The expected value (average) of 1.4 dilutes failures in rare situations (compare performance by situation in 2.8). So evaluation is split by type of situation, and closed-loop evaluation — in which the model’s decisions move the vehicle and the results become the next inputs, as in simulation — checks whether it can actually drive without collisions.

③ Guarding a learned part with rules

In practice, a learned planner is often combined with a rule-based monitor (a safety envelope) such as the RSS of 4.5. If, in hard situations, the learned planner outputs a dangerous trajectory with probability 1%, and the monitor catches and stops it with probability 90%, then if the two are independent, the probability that a dangerous trajectory is actually executed is

$$0.01 \times (1 - 0.9) = 0.001$$

— a tenth. But if the monitor uses the same perception results, the two are not independent. If 30% of the learned planner’s errors come from “perception missed an object”, and the monitor cannot see that object either, then

$$0.01 \times 0.3 + 0.01 \times 0.7 \times 0.1 = 0.003 + 0.0007 = 0.0037$$

about four times the estimate that assumed independence. It is the same problem as the common causes of 1.1. It is important for the monitor to use separate sources of information wherever possible (other sensors, or simpler detection).

Summary

The modular approach builds the system in parts, making it easier to test each part and pinpoint causes. The end-to-end approach trains everything from input to driving together, discarding no information but making the inside hard to see. Most real systems are in between
In imitation learning, drifts into unseen states tend to compound. Average errors (open-loop evaluation) dilute failures in rare hard situations, so check with evaluation by situation and with closed-loop evaluation
Guarding a learned part with rules reduces danger, but sharing the same information source weakens the effect through common causes. The architecture changes the unit of validation and how safety evidence is presented

Quick quiz

Q1. What does “information is lost between parts” mean in a modular system? Give an example.

Something perception did not put in the object list (unfamiliar debris, a pothole) never reaches the planning part, even if it appears in the image. Information that does not fit the hand-over format becomes invisible to later parts.

Q2. In ①, with a drift probability of ε = 0.002 and T = 500 decisions, what is the probability of never drifting?

(1 − 0.002)⁵⁰⁰ ≈ e^(−0.002 × 500) = e^(−1) ≈ 0.37.

Q3. In ②, if the share of hard situations rises to 10%, what is the average error?

0.9 × 0.2 + 0.1 × 4 = 0.18 + 0.40 = 0.58 m.

Q4. In ③, why does the “0.01 × 0.1” estimate become too optimistic if the monitor uses only the same perception results as the learned planner?

Because a perception miss is a common cause that affects the learned planner and the monitor at the same time. The monitor cannot see the object either, so it cannot notice, and the multiplication that assumed independence no longer holds.