Part 3 · Deep Learning & AI

3.2 Backpropagation

Passing errors backward

10-minute readPrerequisites: 3.1 Neural networks, 2.3 Gradient descent▶ Video (about 5 min)

What you will learn

Explain backpropagation as a way of computing gradients by passing the output error back to each weight, layer by layer from the end
Use the chain rule (multiplying slopes) to compute the gradients of a small network
Explain why it is fast, and problems such as vanishing and exploding gradients

Build intuition

The weights of the neural networks in 3.1 are learned with the gradient descent of 2.3. That requires, for every single weight, its slope: how much the loss changes when that weight moves a little. But there are millions to billions of weights. Nudging them one at a time and recomputing the whole network after each nudge would take millions of calculations for a single step.

So we work backward from the output. First compute how wrong the output was (the error), then pass it to the layer before, as “this value in this layer was this much to blame”. Each layer passes the error it receives further back, and on the way works out the slopes of its own weights. In this way, one backward pass produces the slopes of all the weights. That is backpropagation.

It works like a train of gears. If gear A turning once makes gear B turn twice, and B turning once makes C turn three times, then turning A once turns C 2 × 3 = 6 times. Multiplying the transmission ratios along the way tells you how a change travels from one end to the other.

A tiny network with just two weights, w₁ and w₂. The top row (forward pass) computes from the input to the loss; the bottom row (backward pass) carries the slope of the loss from right to left by multiplication, giving the slopes of all the weights.

Check with math

① The chain rule: slopes travel by multiplication

When the loss L depends on the prediction ŷ, ŷ depends on h, and h depends on the weight $w_1$, the rate at which L changes as $w_1$ moves a little is the product of the rates along the way. This is the chain rule — the same as multiplying gear ratios.

$$\frac{\partial L}{\partial w_1} = \frac{\partial L}{\partial \hat{y}} \cdot \frac{\partial \hat{y}}{\partial h} \cdot \frac{\partial h}{\partial w_1}$$

② Working through the figure’s example

Forward pass: from input x = 1 and $w_1 = 0.5$, $h = \mathrm{ReLU}(0.5) = 0.5$; from $w_2 = 2$, $\hat{y} = 1.0$. The answer is y = 2, so the loss (squared error, 2.2) is $L = (1.0 - 2)^2 = 1.0$.

Backward pass: work out the slopes from the end:

$\partial L / \partial \hat{y} = 2(\hat{y} - y) = -2$ (the error says “the prediction is too small”)
Since $\hat{y} = w_2 h$: $\partial L / \partial w_2 = -2 \times h = -1$ and $\partial L / \partial h = -2 \times w_2 = -4$
Since $h = \mathrm{ReLU}(w_1 x)$, and ReLU has slope 1 where z is positive (3.1): $\partial L / \partial w_1 = -4 \times 1 \times x = -4$

Taking one update step with learning rate η = 0.1 (2.3): $w_1 \leftarrow 0.5 - 0.1 \times (-4) = 0.9$ and $w_2 \leftarrow 2 - 0.1 \times (-1) = 2.1$. Recomputing gives h = 0.9 and ŷ = 1.89, and the loss drops from 1.0 to $(1.89 - 2)^2 \approx 0.012$. Not only $w_2$, next to the output, but also $w_1$, further back, has moved in proportion to “how much it was to blame”.

③ Why it is fast

Measuring each slope by nudging one weight at a time (numerical differentiation) gives the same answers. Increasing $w_1$ by 0.001, for example, lowers the loss from 1.000 to about 0.996, a slope of about −4. But with N weights this approach needs about N recomputations of the network for every step. Backpropagation delivers all the slopes in one forward pass and one backward pass (the cost of about two or three forward passes), however many weights there are. With 100 million weights, that is the difference between 100 million computations and a handful. Numerical differentiation is still used to check, on a few weights, that an implementation is correct.

④ Vanishing and exploding gradients

As the network gets deeper, the chain-rule product gets longer. If the factors being multiplied are less than 1, the slopes shrink for layers further back. The sigmoid function’s slope, for example, is at most 0.25, so ten layers’ worth gives $0.25^{10} \approx 0.000001$ — about one millionth. The weights in the early layers barely move and learning stalls. This is the vanishing gradient problem. If the factors are greater than 1, the slopes grow too large, as in $1.5^{10} \approx 58$, and learning goes wild (exploding gradients).

Countermeasures include using ReLU, which has a stretch with slope 1, choosing the initial weights carefully, evening out the size of values partway through the network (normalization), adding shortcuts that skip over layers (residual connections), and clipping slopes that are too large. It was the accumulation of such tricks that made deep networks trainable.

Summary

Backpropagation passes the output error back layer by layer, obtaining the slopes of all the weights in a single backward pass
It rests on the chain rule: slopes (rates of change) travel as products of the rates along the way. Gradient descent then updates the weights using these slopes
In deep networks the product gets long, and gradients can vanish or explode. ReLU, careful initialization, normalization and residual connections help prevent this

Quick quiz

Q1. When gear A turns once, gear B turns three times; when B turns once, C turns 0.5 times. How many times does C turn when A turns once?

3 × 0.5 = 1.5 times. Multiplying the ratios along the way — that is the idea of the chain rule.

Q2. In the figure’s example, what happens to the weight slopes if the answer is y = 1 (with the prediction still ŷ = 1.0)?

The loss is 0 and $\partial L / \partial \hat{y} = 2(1.0 - 1) = 0$, so every slope is 0 and the weights do not move. If the prediction matches the answer, there is nothing to fix.

Q3. For a network with 10 million weights, how many times per step must the network be computed if each slope is measured by nudging one weight at a time? And with backpropagation?

Nudging one at a time takes about 10 million computations. Backpropagation gets all the slopes from one forward pass and one backward pass.

Q4. In a 20-layer network, suppose the slope is multiplied by 0.5 at each layer on its way back. How large is the slope that reaches the first layer, compared with the output side?

$0.5^{20} \approx 0.000001$ — about one millionth. The first layer’s weights barely move and learning stalls (vanishing gradients).