Part 4 · How Automated Driving Works

4.3 Sensors and fusion

Strengths and weaknesses of cameras, LiDAR and radar

10-minute readPrerequisites: 4.2 System overview, 1.3 Probability distributions (the normal distribution), 1.4 Expected value and variance▶ Video (about 5 min)

What you will learn

Explain what sensors such as cameras, LiDAR and radar measure, and what each is good and poor at
Check with formulas how distance is measured from the round trip of light and radio waves, and why a stereo camera’s distance error grows with range
Calculate sensor fusion, which combines several sensors according to the size of their errors, and explain its safety caveats

Build intuition

At the very start of the flow in 4.2 are the sensors. Anything the sensors fail to capture cannot be rescued by perception or planning downstream. An automated vehicle carries several kinds of sensor with quite different characters.

Sensor	What it measures	Good at	Poor at
Camera	Images of light (appearance)	Telling colours, shapes and text apart (traffic lights, signs, lane markings, pedestrians vs cyclists). Cheap	Distance (especially far away), darkness, low sun and sudden changes in brightness, raindrops or dirt on the lens
LiDAR	Round-trip time of laser light → a set of 3D points (a point cloud)	Accurate distance and shape; works the same by day or night	Cannot see colour or text. Rain, fog and snow scatter the light. Weak returns from black or distant objects. Expensive
Radar	Round-trip time and frequency shift of radio waves → distance and relative speed	Measures relative speed directly; long range; copes well with rain, fog and darkness	Coarse lateral resolution, poor at telling shapes apart; distinguishing stationary objects and metal reflections such as manhole covers or overhead signs
Ultrasonic	Round-trip time of sound	Close range within a few metres (parking)	Long range, high speeds
GNSS / IMU	Satellite signals / acceleration and rotation	The vehicle’s own position and motion	Tunnels and urban canyons (GNSS); drift that builds up over time (IMU). → 4.4 Localization and maps

No single kind of sensor can do everything. Combining information from several sensors so that each covers for the others’ weaknesses is sensor fusion. For example, when measuring the position of the car ahead, a camera knows its “lateral position” precisely from where it appears in the image, but its distance only vaguely. Radar is the reverse: precise in distance but coarse laterally. Putting the two together gives a position that is precise in both directions.

How the errors spread when a camera and a radar estimate the position of the car ahead (schematic, sizes exaggerated). Combining sensors that are good in different directions shrinks the region of the estimate.

There are two broad ways to combine them: overlaying the sensors’ raw data (early fusion), or detecting objects separately with each sensor and then merging the resulting lists (late fusion). Early fusion uses all the information without discarding any, but a disturbance in one sensor more easily disturbs the whole; late fusion is easier to check sensor by sensor, but information each sensor discarded cannot be recovered.

Check with math

① Measuring distance from the round trip of light and radio waves

LiDAR measures the time t from sending a laser pulse until it bounces off an object and returns. With c the speed of light (about $3 \times 10^8$ m/s, 300,000 km per second), and since the light travels there and back,

$$d = \frac{c\,t}{2}$$

For an object 100 m away, t = 2 × 100 ÷ (3 × 10⁸) ≈ 0.67 μs (0.67 millionths of a second). A timing error of just 1 ns (a billionth of a second) shifts the distance by 15 cm. The technology for timing extremely short intervals precisely is what makes LiDAR distances so accurate.

Radar also measures distance from the round trip of radio waves. In addition, waves reflected from a moving object come back with a slightly shifted frequency (the Doppler effect), so relative speed can be measured directly from the shift. For a 77 GHz radar and a relative speed of 10 m/s, the shift is about 5 kHz. This differs from cameras and LiDAR, which calculate speed afterwards from changes in position.

② Cameras struggle with distance, more so far away

A single camera image cannot measure distance directly. A stereo camera, with two cameras side by side, finds distance Z from how far the same object is shifted between the left and right images (the disparity d, in pixels):

$$Z = \frac{f\,B}{d}$$

Here f is the focal length (expressed in pixels) and B the spacing between the two cameras. With f = 1000 pixels and B = 0.3 m, an object 30 m away has a disparity of 1000 × 0.3 ÷ 30 = 10 pixels. If the disparity is read with an error Δd, the derivative of 0.2 gives the distance error

$$\Delta Z \approx \frac{Z^2}{f\,B}\,\Delta d$$

which is proportional to the square of the distance. For Δd = 0.5 pixels:

Distance Z	15 m	30 m	60 m
Disparity d	20 pixels	10 pixels	5 pixels
Distance error ΔZ	about 0.4 m	about 1.5 m	about 6 m

Doubling the distance roughly quadruples the error. This is why radar and LiDAR are better than cameras at the distance of far-off objects. AI that estimates distance from a single camera also infers it from an object’s size and position in the image, so its error too grows with range.

③ Fusion: give more weight to the smaller error

Suppose LiDAR measures the distance to an object as 50.0 m (error standard deviation $\sigma_1 = 0.3$ m) and radar as 50.6 m ($\sigma_2 = 0.6$ m). Rather than trusting one of them, use both. If the errors are independent and follow normal distributions (1.3), the most likely value is the average weighted by the inverse of each variance:

$$\hat{x} = \frac{x_1/\sigma_1^2 + x_2/\sigma_2^2}{1/\sigma_1^2 + 1/\sigma_2^2}, \qquad \frac{1}{\hat{\sigma}^2} = \frac{1}{\sigma_1^2} + \frac{1}{\sigma_2^2}$$

The weights are 1/0.09 : 1/0.36 = 4 : 1, so x̂ = (4 × 50.0 + 1 × 50.6) ÷ 5 = 50.12 m. The combined error is σ̂ ≈ 0.27 m, smaller even than the better sensor, LiDAR (0.3 m). Weighting the precise sensor heavily and the vague one lightly, and adding up the information from both, is the basic idea of fusion. Repeating the same calculation at every time step, combining the prediction of ③ in 4.2 with each new measurement, is the Kalman filter, widely used for tracking and localization.

When different sensors are good in different directions, as in the figure, this weighting works direction by direction: the lateral position ends up close to the camera’s value and the distance close to the radar’s. This calculation, however, assumes that each σ is correct. An overconfident sensor that underestimates its σ drags the whole result towards its wrong value (3.7).

Summary

Cameras are good at telling things apart but poor at distance and changes in brightness; LiDAR is accurate in distance and shape but weak in fog and rain; radar is strong on speed and bad weather but poor at telling shapes apart. No single kind of sensor can do everything
Distance is found from the round-trip time of light or radio waves (d = ct/2) or from a stereo camera’s disparity (Z = fB/d). A stereo camera’s distance error grows in proportion to the square of the distance
Fusion combines measurements weighted by the inverse of their variances, making the error smaller than any single sensor’s. It presumes combining sensors that work on different principles, and checking that each σ is right and that the sensors are aligned in position and time

Quick quiz

Q1. At night, a vehicle drives out of a tunnel straight into strong low sun. Which sensor is likely to be weakened, and what can cover for it?

The camera (it is weak against low sun and sudden changes in brightness). LiDAR and radar send out their own light or radio waves, so they are hardly affected by the surrounding brightness and can cover for it.

Q2. A LiDAR pulse takes 0.4 μs to return. How far away is the object?

d = 3 × 10⁸ × 0.4 × 10⁻⁶ ÷ 2 = 60 m.

Q3. With the stereo camera in the text (f = 1000 pixels, B = 0.3 m, disparity error 0.5 pixels), roughly what is the distance error for an object 90 m away?

ΔZ ≈ 90² ÷ (1000 × 0.3) × 0.5 = 8100 ÷ 300 × 0.5 = 13.5 m — nine times the error at 30 m (three times the distance, so 3² = 9 times the error).

Q4. Two sensors, each with σ = 0.4 m, measure the same distance. What is the error after combining them?

1/σ̂² = 1/0.16 + 1/0.16, so σ̂ = 0.4 ÷ √2 ≈ 0.28 m — the same form as the standard error σ/√n of 1.5. This holds only when the two errors are independent.