Part 4 · How Automated Driving Works

4.2 System overview

Perception, planning and control

10-minute readPrerequisites: 4.1 Levels of automation and the ODD, 1.4 Expected value and variance (standard deviation)▶ Video (about 5 min)

What you will learn

Explain an automated driving system as the flow sensors → perception → planning → control, together with the monitoring that watches over it
Calculate how processing latency affects the distance needed to stop
Estimate with simple formulas how errors in one part pass on to later stages and how prediction errors widen further ahead, and explain the idea of allocating safety requirements to the parts

Build intuition

When people drive, they take in their surroundings with their eyes and ears (perception), decide how to move (planning), and operate the steering wheel and pedals (control). An automated driving system follows the same basic flow. The differences are that it uses sensors such as cameras and radar in place of eyes, and that it repeats the flow tirelessly, from ten to several dozen times a second.

The basic flow of an automated driving system (schematic). It repeats measure, understand, decide and act in a fast cycle, while monitoring watches over the whole.

The role of each stage and the main techniques used in it are as follows:

Stage	Role	Main techniques	More in
Sensors	Measure the surroundings and the vehicle’s own state	Cameras, LiDAR, radar, GNSS (satellite positioning), IMU (measuring acceleration and rotation), wheel rotation	4.3 Sensors and fusion
Perception	Work out what is where and how it is moving; work out where the vehicle itself is	Object detection and segmentation, tracking, sensor fusion, localization and maps	3.4, 4.4 Localization and maps
Planning	Predict how others will move and decide how to drive	Predicting the motion of surrounding vehicles and pedestrians, choosing behaviour (stop, change lanes), planning the path and speed	4.5 Prediction and motion planning
Control	Move the vehicle along the chosen path at the chosen speed	Control of acceleration and steering; brakes and steering operated by electrical signals (by-wire)	0.2 (braking distance)
Monitoring	Watch whether everything is working correctly and whether the vehicle is inside its ODD	Fault detection, ODD monitoring, driver monitoring, fallback	4.1, 4.7 Three pillars of safety

Between stages, information is handed over in fixed forms. Perception passes on “a list of objects (type, position, speed, size, confidence)” and “where it is possible to drive”; planning passes on “the trajectory for the next few seconds (position and speed at each moment)”. Because these hand-over formats (interfaces) are clearly defined, each stage can be built, tested and checked for performance on its own. This way of building the system from parts is called the modular approach. How it differs from the end-to-end approach, which trains the whole system as one large AI, is covered in 4.6.

Machine learning is used mainly in perception and prediction, where the possible conditions are endless. Control, by contrast, centres on methods based on physical models of vehicle motion, and its behaviour is comparatively easy to check by calculation. A single system thus contains parts of quite different kinds side by side.

Check with math

① Latency lengthens the stopping distance

From the moment the sensors capture a scene until the brakes actually start to act, each stage takes time to process. This is called latency. A breakdown might look like this (an illustrative example):

Stage	Latency
Sensors (capture and transmit)	50 ms
Perception	80 ms
Planning	50 ms
Control (from command until the brakes start to act)	120 ms
Total	300 ms (0.3 s)

1 ms (millisecond) is a thousandth of a second. The vehicle keeps moving at the same speed during the delay, so the stopping distance is the braking distance of 0.2 plus “the distance travelled during the delay”:

$$s_{\text{stop}} = v\,t_{\text{delay}} + \frac{v^2}{2a}$$

At 100 km/h (about 27.8 m/s) with deceleration $a = 5\ \mathrm{m/s^2}$, the braking distance is 27.8² ÷ 10 ≈ 77 m. Adding the roughly 8 m travelled during the 0.3-second delay gives about 86 m. A person usually takes around a second from noticing to pressing the brake, which by the same calculation gives about 105 m. The system’s delay is shorter than a person’s, but it is not zero: every extra 0.1 s of delay adds about 2.8 m to the stopping distance.

Also, if perception runs every 0.1 s (10 Hz), a pedestrian who steps out just after a processing cycle may wait up to 0.1 s for the next one. The processing cycle adds to the latency too.

② Errors add up in series

If any one of perception, prediction, planning or control goes wrong, the vehicle cannot behave correctly. Let $p_1, p_2, \ldots$ be the probabilities that each stage errs in a given situation, and assume they are independent. By the multiplication of 1.1, the probability that the whole system is right is

$$P(\text{all correct}) = (1-p_1)(1-p_2)\cdots(1-p_k) \approx 1 - (p_1 + p_2 + \cdots + p_k)$$

When every p is small, the error probability can be estimated almost by addition, as on the right. With perception 0.1%, prediction 0.2%, planning 0.05% and control 0.01%, the overall error is about 0.36% (0.36% by multiplication as well). Most of the overall error comes from prediction and perception, and however much control is improved, the total hardly changes. The calculation shows where effort should go.

However, when something like heavy rain degrades perception and prediction at the same time (a common cause, as in 1.1), errors concentrate in particular situations. Even if the average figures are small, looking only at rainy days, several stages become likely to err at once. Estimates need to be made separately for each situation (each set of ODD conditions). Moreover, later stages have no way of knowing about an object that an earlier stage missed: a miss in perception can hardly be recovered in planning or control.

③ Prediction errors widen further ahead

Planning predicts where surrounding vehicles will be a few seconds from now. The simplest prediction is “current position + speed × time”. If the position measured by perception has an error (standard deviation $\sigma_p$) and the speed has an error ($\sigma_v$), and the two are independent, the error of the predicted position t seconds ahead is

$$\sigma(t) = \sqrt{\sigma_p^2 + (\sigma_v\,t)^2}$$

Independent errors are not simply added: they are squared, added, and then the square root is taken — the same form as combining two standard errors in 1.7. With $\sigma_p = 0.3$ m and $\sigma_v = 0.5$ m/s:

Seconds ahead	0	1	2	3
Prediction error σ(t)	0.30 m	0.58 m	1.04 m	1.53 m

Three seconds ahead, σ ≈ 1.5 m, so ±2σ (about a 95% range) is ±3 m — close to the width of a lane (about 3.5 m). Because predictions become vaguer further ahead, planning leaves bigger margins for later moments and replans each time a new measurement arrives. If perception also passes on how large its errors are, planning can adjust its margins accordingly (the uncertainty of 3.7). In reality, surrounding vehicles may also turn or stop, so the errors are larger still. → 4.5 Prediction and motion planning

Summary

An automated driving system measures with sensors, works out its surroundings and its own position in perception, predicts and decides how to drive in planning, and moves the vehicle in control. It repeats this flow many times a second, while monitoring watches over the whole
The latencies of the stages add up, and the vehicle keeps moving during the delay (stopping distance = speed × delay + braking distance). Errors in the parts add up in series, so the overall error is roughly the sum of the stages’ errors
Prediction errors widen further ahead (independent errors are squared and added). Allocating the overall safety requirement to the parts, and passing uncertainty on to later stages, are important

Quick quiz

Q1. “The car ahead suddenly slowed down, so the brakes were applied harder.” Split this into perception, planning and control.

Perception: measure the position and speed of the car ahead and recognize that its speed has dropped sharply. Planning: predict that the vehicle will get too close and decide how much to slow down. Control: apply the brakes to achieve the chosen deceleration.

Q2. If the total latency rises from 0.3 s to 0.5 s, how many metres longer is the stopping distance at 100 km/h?

The delay grows by 0.2 s, so the distance grows by 27.8 × 0.2 ≈ 5.6 m (from about 86 m to about 91 m).

Q3. In the example in the text (σ_p = 0.3 m, σ_v = 0.5 m/s), if better speed measurement brings σ_v down to 0.25 m/s, what is the prediction error 3 seconds ahead?

$\sqrt{0.3^2 + (0.25 \times 3)^2} = \sqrt{0.09 + 0.5625} \approx 0.81$ m — almost half of the 1.53 m. For predictions further ahead, the speed error has the larger effect.

Q4. To reduce the overall error in example ②, which helps more: cutting the control error (0.01%) to a tenth, or halving the prediction error (0.2%)?

Halving the prediction error. The overall error of about 0.36% becomes about 0.351% if control improves, but about 0.26% if prediction is halved. Start with the stage that contributes the largest error.