Part 2 · Machine Learning Basics

2.3 Gradient descent

Walking downhill to find the best solution

10-minute readPrerequisites: 2.2 Linear regression and loss functions, 0.2 Reading math notation (derivatives)▶ Video (about 5 min)

What you will learn

Explain gradient descent as “reading the slope and taking small steps downhill”
Use the update formula to move a parameter by one step
Explain how the size of the learning rate matters, and why mini-batches, local minima and randomness make training results vary

Build intuition

Imagine searching for the bottom of a valley on a mountain in thick fog. You cannot see far, but you can feel which way the ground slopes under your feet. If you keep taking a step in the direction that goes downhill, you will eventually reach the bottom. That is gradient descent.

In machine learning, the height of the mountain is the loss (the MSE from 2.2), and your position on the map is the set of parameters. We search for the position where the loss is smallest, using the slope — the gradient — as our guide. The line in 2.2 could be found in one step with a formula, but this method also works for models such as neural networks that no formula can solve.

Take the reaction-distance example from 2.2, fix the intercept at b = 0.47, and search for the slope a alone. To keep the numbers tidy, speed is expressed in units of 10 km/h (40 km/h becomes 4.0). The best slope is then about 2.0.

The loss (MSE) as the slope a changes, and the positions reached by gradient descent with learning rate η = 0.02 (black dots, up to step 5). Where the slope is steep it moves a long way; near the bottom the slope flattens, so the steps shrink naturally. The intercept b is fixed at its value from 2.2 (0.47).

At the start, a = 0, the slope is steep, so a single step takes us all the way to a ≈ 1.28. Closer to the bottom the slope flattens and the steps shrink; after five steps we have almost settled at the bottom, a ≈ 1.98.

Check with math

① The update formula

$$\theta \leftarrow \theta - \eta\,\frac{\partial L}{\partial \theta}$$

The arrow ← means “replace the left side with the value on the right” (see Notation). $\partial L / \partial \theta$ is the rate at which the loss L changes when θ is moved a little — the slope (the partial derivative from 0.2). If the slope is positive (uphill to the right), θ decreases; if it is negative (downhill to the right), θ increases — so we always move downhill. η (eta) is the learning rate, which sets the size of each step.

② Working through the example one step at a time

Differentiating the MSE with respect to a gives the slope:

$$\frac{d\,\mathrm{MSE}}{da} = -\frac{2}{n}\sum_{i=1}^{n} x_i\,(y_i - a x_i - b)$$

At a = 0 this is about −64.2 (a steep downhill slope). With learning rate η = 0.02:

$$a \leftarrow 0 - 0.02 \times (-64.2) \approx 1.28$$

At the new a = 1.28 the slope is about −22.9, so a ← 1.28 + 0.02 × 22.9 ≈ 1.74 — and so on. With several parameters, we line up their partial derivatives into the gradient $\nabla L$ (0.2) and move all the parameters at once:

$$\boldsymbol{\theta} \leftarrow \boldsymbol{\theta} - \eta\,\nabla L$$

③ Choosing the learning rate

Starting from the same a = 0 and changing only the learning rate (the best value is a ≈ 1.99):

Learning rate η	After 1 step	After 2	After 3	After 10	Behavior
0.005	0.32	0.59	0.82	1.65	Too small; barely moves
0.02	1.28	1.74	1.90	1.99	About right
0.05	3.21	1.25	2.45	1.98	Overshoots, swinging from side to side as it closes in
0.07	4.49	−1.14	5.92	−17.1	The overshoot keeps growing (divergence)

If the learning rate is too large, each step jumps across the valley to the opposite slope, so training swings back and forth or runs further and further away (diverges). A rate that is too small is safe but slow. In practice, a common trick is to start fairly large and reduce the rate gradually.

④ Mini-batches (stochastic gradient descent)

When the data run to millions of images, computing the slope over all of them for every step is far too heavy. Instead, each step estimates the slope from a small, randomly chosen subset — a mini-batch, say 32 images. This is called stochastic gradient descent (SGD). The slope estimates are noisy, but each step is cheap, so overall the bottom is reached faster. One pass through the whole dataset is counted as one epoch.

⑤ Landscapes with many valleys (local minima)

The loss of linear regression is a clean bowl with a single bottom. The loss of a neural network, however, is a complicated landscape full of valleys and flat stretches. A valley that is lowest in its neighborhood but not necessarily lowest overall is called a local minimum. Which valley training settles into depends on the starting point (the initial parameter values) and on how the mini-batches are drawn (randomness). As a result, training on the same data with the same settings can produce slightly different models.

Summary

Gradient descent reads the slope (gradient) of the loss and moves the parameters downhill a little at a time, searching for where the loss is smallest
The update is “θ ← θ − η × slope”. A learning rate η that is too small is slow; one that is too large swings or diverges
With large datasets, steps use mini-batches (SGD). In complex models, initial values and randomness change the result, so record the setup and check across several training runs

Quick quiz

Q1. If the slope of the loss, ∂L/∂θ, is positive (uphill to the right), should θ increase or decrease?

Decrease. When the ground rises to the right, moving left (decreasing θ) lowers the loss. The minus sign in the update formula sets this direction.

Q2. With θ = 2, a slope of 4 and a learning rate η = 0.1, what is θ after one step?

θ ← 2 − 0.1 × 4 = 1.6.

Q3. After training starts, the loss grows larger and larger instead of falling. What should you suspect first?

A learning rate that is too large. Each step leaps across the valley to a higher spot on the other side, so training diverges. Try a smaller learning rate.

Q4. Training twice on the same data with the same settings produced models with detection rates of 91.2% and 90.5%. Why? What should you do for safety?

The initial parameter values and the way mini-batches are drawn are set by random numbers, so training settles into slightly different valleys. Record which data, settings and random seed were used, check the variation across several training runs, and include that variation in the evaluation.