Part 2 · Machine Learning Basics

2.4 Logistic regression

Classifying with probabilities

10-minute readPrerequisites: 2.2 Linear regression and loss functions, 2.3 Gradient descent▶ Video (about 5 min)

What you will learn

Explain how logistic regression answers classification questions with a probability (the sigmoid function)
Calculate the classification loss, cross-entropy, and explain why squared error is not used
Explain how the threshold that turns a probability into a decision sets the balance between misses and false alarms

Build intuition

Suppose that on a motorway you want to predict whether a car in the next lane will cut into your lane. The clue is its lateral speed towards you. Recording 18 situations shows that cars drifting towards you faster cut in more often — but no clean line can be drawn. One car cut in at 0.35 m/s, while another at 0.7 m/s did not.

The answer is one of two values: “cuts in” (1) or “does not cut in” (0). Fitting a straight line as in 2.2 would give predictions above 1 or below 0, which make no sense. Instead we use an S-shaped curve that squeezes any value into a probability between 0 and 1. That is logistic regression.

Eighteen situations (red: cut in; black: did not) and the learned S-shaped curve (the probability of cutting in). Speeds to the right of where the probability crosses the 0.5 threshold (a lateral speed of 0.56 m/s) are classified as “cuts in”. The numbers are an illustrative example.

On the learned curve, the probability of cutting in is about 5% at a lateral speed of 0.2 m/s, about 57% at 0.6 m/s and about 97% at 1.0 m/s. Classifying everything to the right of the point where the probability passes 0.5 (0.56 m/s) as “cuts in” turns this into a classifier. The “0.5” is called the threshold. As we will see, the threshold is sometimes moved for safety.

Check with math

① Turning a score into a probability with the sigmoid function

$$p = \sigma(z) = \frac{1}{1 + e^{-z}}, \qquad z = w x + b$$

z is the same straight-line expression as in 2.2 (a weighted sum when there are several inputs). The sigmoid function σ(z) keeps the result between 0 and 1 whatever z is: z = 0 gives 0.5, larger z moves towards 1 and smaller z towards 0. Training gave w ≈ 7.96 and b ≈ −4.49. At a lateral speed of 0.6 m/s, z = 7.96 × 0.6 − 4.49 ≈ 0.29, so $p = 1/(1 + e^{-0.29}) \approx 0.57$.

② What the weight means (odds)

Running the sigmoid function backwards gives this relationship:

$$\ln\frac{p}{1-p} = w x + b$$

$p/(1-p)$ is called the odds — how many times more likely the event is to happen than not. At heart, logistic regression says that the logarithm of the odds is a straight line in x. The weight w ≈ 7.96 reads as “every extra 0.1 m/s of lateral speed multiplies the odds of a cut-in by $e^{0.796} \approx 2.2$”. As with linear regression, people can check what the learned weights mean.

③ The loss function: cross-entropy

$$L = -\big[\,y \ln p + (1 - y)\ln(1 - p)\,\big]$$

This is called cross-entropy. When the answer is y = 1, only −ln p remains: 0.11 for p = 0.9, 0.51 for p = 0.6 and 2.30 for p = 0.1. Its defining feature is that the more confidently wrong a prediction is, the more steeply the penalty grows. The average over our 18 situations was about 0.37.

Minimizing the total cross-entropy is the same as maximizing the probability that the data turn out exactly as observed (the likelihood):

$$\prod_{i=1}^{n} p_i^{\,y_i}\,(1 - p_i)^{1 - y_i}$$

Taking the ln and adding a minus sign turns this into the total cross-entropy. This approach is called maximum likelihood estimation. Logistic regression has no one-step formula like the one in 2.2, so its parameters are found with the gradient descent of 2.3.

④ Choosing the threshold

The same model gives different decisions at different thresholds (8 of the 18 situations were cut-ins):

Threshold	Boundary speed	Cut-ins predicted	Misses	False alarms
0.7	0.67 m/s	5 / 8	3	1 / 10
0.5	0.56 m/s	6 / 8	2	2 / 10
0.3	0.46 m/s	7 / 8	1	2 / 10

Lowering the threshold reduces misses but generally increases false alarms (with so little data here, lowering it from 0.5 to 0.3 happened to leave the false alarms at two). Where to put the threshold is a design decision, based on how dangerous a miss is compared with a false alarm.

⑤ Three or more classes (the softmax function)

To sort into three or more classes, such as pedestrian, cyclist and car, compute a score $z_k$ for each class k and turn the scores into probabilities with the softmax function:

$$p_k = \frac{e^{z_k}}{\sum_{j} e^{z_j}}$$

Every probability is at least 0 and they add up to 1 (for example, pedestrian 0.7, cyclist 0.2, car 0.1). Image-recognition neural networks perform this calculation in their final stage too.

Summary

Logistic regression classifies by turning the straight line z = wx + b into a probability between 0 and 1 with the sigmoid function. Its weights can be read as multipliers on the odds
The loss is cross-entropy, which penalizes confident mistakes heavily. There is no one-step formula, so it is trained with gradient descent
The threshold that turns probabilities into decisions sets the balance between misses and false alarms — a safety design choice. For three or more classes, use the softmax function

Quick quiz

Q1. What is the value of the sigmoid function σ(z) when z = 0?

$1/(1 + e^{0}) = 1/2 = 0.5$. The point where z = 0 is the dividing line at probability 0.5.

Q2. In a situation where the answer is “cuts in” (y = 1), model A predicts p = 0.9 and model B predicts p = 0.6. What is each model’s cross-entropy?

A: −ln 0.9 ≈ 0.11; B: −ln 0.6 ≈ 0.51. Model A, which gave the correct outcome a higher probability, has the smaller loss.

Q3. If the threshold is lowered from 0.5 to 0.3, what generally happens to misses and false alarms?

More situations are classified as “cuts in”, so misses decrease and false alarms increase. Which to favor depends on how dangerous each kind of error is.

Q4. For a special-purpose vehicle with a shape absent from the training data, the model outputs “probability of not cutting in: 0.99”. Can you trust it?

Trusting it as it stands is dangerous. A model’s probabilities are only reliable in situations similar to its training data. With unfamiliar objects it can be confidently wrong, so the design needs to check by other means or behave on the cautious side.