Part 2 · Machine Learning Basics

2.2 Linear regression and loss functions

Measuring the “gap” in predictions

10-minute readPrerequisites: 2.1 What is machine learning?, 1.4 Expected value and variance▶ Video (about 5 min)

What you will learn

Explain how linear regression (predicting with a straight line) works, and what the slope and intercept mean
Calculate a loss function (such as the mean squared error) that turns the prediction gaps — the residuals — into one number
Explain the difference between squared and absolute error, and the dangers of outliers and of predicting outside the data (extrapolation)

Build intuition

The distance a car travels between the moment the driver notices a hazard and the moment braking begins is called the reaction distance. Eight drivers were each measured at a different speed; the dots in the figure below show the results. The faster the speed, the longer the reaction distance.

If this relationship can be expressed as one straight line, we can predict the reaction distance even at speeds we did not measure. That is linear regression. The question is which line is best. The dots do not sit exactly on any line; they scatter above and below it. We choose the line that makes these gaps — the residuals — smallest overall.

Reaction distances of eight drivers (dots) and the line drawn to make the gaps smallest. The dotted line from each dot to the line is its prediction gap (residual). The numbers are an illustrative example.

The best line turned out to be $\hat{y} \approx 0.20x + 0.47$ (x is the speed in km/h and $\hat{y}$ the predicted reaction distance in m). The slope of 0.20 means “every 1 km/h of extra speed adds about 0.2 m of reaction distance”, which corresponds to a reaction time of about 0.72 seconds (0.20 × 3.6). A strength of linear regression is that people can read the meaning of what it has learned.

Check with math

① The linear regression model

$$\hat{y} = a x + b$$

a is the slope and b the intercept; together they are the parameters θ from 2.1. With several inputs (speed, road condition, driver age and so on), each is multiplied by a weight w and the results are added up. Using the dot product from 0.2, this can also be written $\mathbf{w} \cdot \mathbf{x} + b$.

$$\hat{y} = w_1 x_1 + w_2 x_2 + \dots + w_d x_d + b$$

② Loss functions: turning the gaps into one number

Let the residual of data point i be $e_i = y_i - \hat{y}_i$ (answer − prediction). Adding the residuals as they are lets positives and negatives cancel out. So we square them and take the average; this is the mean squared error (MSE).

$$\mathrm{MSE} = \frac{1}{n}\sum_{i=1}^{n}(y_i - \hat{y}_i)^2$$

For the line in the figure, MSE ≈ 0.81 m². Because m² is hard to interpret, its square root, the RMSE ≈ 0.90 m, is often used as “the typical prediction gap”. Trying other lines, $\hat{y} = 0.25x$ gives MSE ≈ 3.3 and $\hat{y} = 0.20x + 1.0$ gives MSE ≈ 1.1 — both larger. A single number expressing how bad the predictions are is called a loss function. It has the same form as the variance in 1.4, which squares and averages the deviations from the mean.

③ Least squares: the best line has a formula

The a and b that minimize the MSE can be computed directly (least squares):

$$a = \frac{\sum_{i}(x_i - \bar{x})(y_i - \bar{y})}{\sum_{i}(x_i - \bar{x})^2}, \qquad b = \bar{y} - a\bar{x}$$

For our data, $\bar{x} = 38.1$ and $\bar{y} = 8.08$, so a = 248.6 ÷ 1246.9 ≈ 0.199 and b = 8.08 − 0.199 × 38.1 ≈ 0.47. Only simple models such as linear regression can be solved in one step like this, though. Complex models such as neural networks are searched step by step, using 2.3 Gradient descent.

④ Squared vs absolute error: robustness to outliers

The average of the absolute residuals is called the MAE (mean absolute error).

$$\mathrm{MAE} = \frac{1}{n}\sum_{i=1}^{n}\lvert y_i - \hat{y}_i \rvert$$

With squared error, a gap twice as large is penalized four times as much, so large gaps count heavily. That is usually an advantage, but when there is an outlier — say, from a sensor glitch — the line gets pulled strongly towards it.

What happens when one wrong data point (an outlier) — 20 m at 55 km/h — gets mixed in. The line fitted with squared error (red) is pulled strongly towards the outlier; the line fitted with absolute error barely moves.

Fitting method	Prediction at 50 km/h (no outlier)	Prediction at 50 km/h (with outlier)
Minimize squared error (MSE)	10.4 m	12.3 m (+1.8 m)
Minimize absolute error (MAE)	10.3 m	11.0 m (+0.6 m)

Neither is simply “right”: it is a choice about what to weight heavily. And if the outlier is not a sensor glitch but a dangerous event that really happened, it must not be thrown away (1.9). Look at the data and find out what caused the outlier first.

Summary

Linear regression predicts by expressing the relationship between input and output as a straight line (a weighted sum when there are several inputs). People can read what the slope and weights mean
A loss function turns prediction gaps into one number. The mean squared error (MSE) is the classic one, and its square root, the RMSE, is the typical gap in the original units. For linear regression, the line with the smallest MSE has a formula (least squares)
Squared error is easily pulled by outliers, and choosing a loss is a design choice about what to weight heavily. Do not trust predictions outside the range of the data (extrapolation)

Quick quiz

Q1. The prediction is 12 m and the correct value is 10 m. What are the squared error and the absolute error?

The residual is 10 − 12 = −2 m, so the squared error is 4 m² and the absolute error is 2 m.

Q2. The residuals of three data points are +1 m, −1 m and +2 m. What are the MSE and the RMSE?

MSE = (1 + 1 + 4) ÷ 3 = 2 m², and RMSE = √2 ≈ 1.4 m.

Q3. Why not simply add up the residuals and take their average?

Because positives and negatives cancel out. With residuals of +5 m and −5 m, the average is 0 — yet both predictions are far off. Squaring or taking absolute values counts the size of each gap regardless of its direction.

Q4. Can a straight line fitted to stopping distances at 20–60 km/h be used to predict the stopping distance at 100 km/h?

It should not be. It is a prediction outside the range of the data (extrapolation), and because the stopping distance includes a part that grows with the square of speed (the braking distance), the straight line comes out about 19 m too short. That is an error on the dangerous side.