Part 2 · Machine Learning Basics

2.6 Overfitting and regularization

Preventing “memorizing too much”

10-minute readPrerequisite: 2.2 Linear regression and loss functions▶ Video (about 5 min)

What you will learn

Tell overfitting (memorizing too much) from underfitting (too simple) by comparing errors on training data and on new data
Explain the roles of training, validation and test data, and how validation data are used to choose a model
Explain how regularization (penalizing large weights) and early stopping prevent overfitting

Build intuition

Think of someone who memorized the answers to past exam papers. If the same questions come up, they score full marks — but they cannot answer a question that has been changed slightly. The same thing happens in machine learning. Fitting the training data so closely that performance drops on new data is called overfitting.

Continuing from 2.2, this time we learn the relationship between speed and stopping distance (reaction distance + braking distance) from ten data points. Because stopping distance includes a part that grows with the square of speed, the relationship is a curve. Let us fit three models: degree 1 (a straight line), degree 2 and degree 9. The higher the degree, the more twisting and complex the curves the model can express.

Three models of different degrees fitted to the same ten training points (colored dots). The grey dots are new data not used for training. The numbers above each panel are the errors (MSE, m²) on the training data and on the new data. The numbers are an illustrative example.

The degree-1 line cannot bend, so it misses every point a little (underfitting). The degree-9 curve passes through all ten points, so its error on the training data is 0 — but it swings wildly between the points and, at 77 km/h, predicts a stopping distance of “−49 m”. The degree-2 curve does not pass exactly through the points, yet it does best on new data.

Check with math

① Error on training data vs error on new data

Changing the degree of the model, we compare the two errors (MSE):

Model degree	1	2	3	5	7	9
Error on training data	6.7	5.0	5.0	4.0	2.0	0.0
Error on new data	7.3	4.2	4.3	9.1	79	1010

The more complex the model, the lower the error on the training data keeps going. The error on new data (the generalization error), however, first falls and then starts rising at some point — a U shape. A model that is too simple errs through bias: it cannot express the true shape. A model that is too complex errs through variance: it bends to fit even the random scatter in the data.

Also, no model can genuinely achieve an error smaller than the scatter built into the data itself (about 3.5 m² here). If the training error falls far below that, suspect memorization.

② Splitting the data three ways

Training data: used to learn the parameters
Validation data: used to choose the settings people decide, such as model complexity and learning rate (hyperparameters)
Test data: used once, at the end, to measure the true performance

Using the “new data” in the table above as validation data would lead us to pick degree 2. But choosing again and again while watching the validation data ends up overfitting the validation data instead. That is why the final evaluation uses test data that have never been touched. When data are scarce, cross-validation — repeating the validation with different splits and averaging — is also used.

③ Regularization: penalizing large weights

The degree-9 coefficients turned out to be huge, such as ±958. Large weights mean that a small change in the input moves the prediction a lot, so the curve swings. The fix is to add a penalty on the size of the weights to the loss:

$$L' = \frac{1}{n}\sum_{i=1}^{n}(y_i - \hat{y}_i)^2 + \lambda \sum_{j} w_j^2$$

λ (lambda) sets the strength of the penalty. Penalizing the sum of the squared weights is called ridge regression (L2 regularization); the general idea is regularization. Applying it to the degree-9 model gives:

Penalty strength λ	0	0.001	0.01	0.1	1
Error on training data	0.0	2.6	3.3	4.0	11.1
Error on new data	1010	35	14.6	7.5	12.9

The stronger the penalty, the smoother the curve and the lower the error on new data. Too strong, however, and the model becomes too simple and underfits (λ = 1). λ is chosen with validation data too. Other effective ways to prevent overfitting include early stopping — stopping training when the validation error starts to rise (2.3) — and simply collecting more data.

Summary

Overfitting means fitting the training data so closely that new data suffer. Suspect it when only the training error falls while the error on new data rises
Split data into training, validation and test sets. Choose settings such as complexity with validation data, and use the test data once, at the end
Regularization (penalizing large weights) and early stopping produce smoother models that work on new data. Too strong a penalty leads to underfitting

Quick quiz

Q1. A model scores 99% accuracy on the training data but 75% on the validation data. What is happening?

Overfitting. The model has memorized the training data and does not fit new data. Try a simpler model, stronger regularization, more data or early stopping.

Q2. A model scores 80% on training data and 79% on validation data; both fall short of the 95% target. What should you do?

The gap is small and both scores are low, so this is underfitting. Try a more expressive model, add useful inputs (features) or weaken the regularization.

Q3. What happens if the regularization strength λ is made too large?

The weights are held down too much, the model becomes too simple and it underfits. In the table, λ = 1 made both the training error and the error on new data large.

Q4. Consecutive frames of a video taken at one intersection were randomly assigned to training and test data. What is the problem?

Nearly identical images end up in both sets, so the test score comes out higher than the model’s true ability (data leakage). The data must be split by whole groups, such as location or day.