Part 1 · Probability & Statistics

1.6 Estimation and confidence intervals

Inferring the true value from your data

10-minute readPrerequisite: 1.5 Law of large numbers and the central limit theorem▶ Video (about 5 min)

What you will learn

Explain the difference between a point estimate and an interval estimate
Calculate a 95% confidence interval for a proportion or a mean as “estimate ± 2 × standard error”, and use it to decide whether a requirement is met
Explain how to read a confidence interval correctly, why we divide by n − 1, and what to watch out for when data are scarce

Build intuition

In 1.5 we asked: “If we know the true detection rate is 90%, how much do the observed values vary?” Real tests work the other way round: the true value is unknown, and all you have are the test results.

Suppose a detector detected the target in 180 out of 200 tests. The detection rate is 180 ÷ 200 = 90%. But the true detection rate is not necessarily exactly 90% — it might be 88%, or 92%.

So we answer with a range: “the true value probably lies within this range”. That range is a confidence interval. For this result, the 95% confidence interval is roughly 86% to 94% (calculated below).

Confidence intervals support decisions. With a requirement of “a detection rate of at least 85%”, the lower end of the interval (86%) is above 85%, so we can say with 95% confidence that the requirement is met. Yet with the same 90% from 90 out of 100 tests, the interval widens to 84–96%; its lower end falls below 85%, so we cannot say so yet. The same “90%” supports different conclusions depending on the number of tests.

A simulation: a detector whose true detection rate is 90% goes through “200 tests” twenty times over, and a 95% confidence interval is computed each time. Nineteen intervals contain the true value (dotted line) and one (red) misses. “As a method, it produces intervals that contain the true value about 95% of the time” — that is what a 95% confidence interval means.

Check with math

① Point estimates and interval estimates

Estimating the true value from data as a single number is a point estimate; estimating it as a range is an interval estimate. Estimates are written with a “hat”, as in $\hat{p}$ (see Notation).

$$\hat{p} = \frac{180}{200} = 0.90$$

② 95% confidence interval: estimate ± 2 × standard error

Compute the standard error from 1.5 using the estimate in place of the unknown true value, and take twice that as the half-width (more precisely, 1.96 instead of 2).

$$\hat{p} \pm 2\sqrt{\frac{\hat{p}(1-\hat{p})}{n}} = 0.90 \pm 2\sqrt{\frac{0.90 \times 0.10}{200}} = 0.90 \pm 0.042$$

That gives 86% to 94% (more precisely 85.8% to 94.2%). For a mean the form is the same: with s the standard deviation of the data, $\bar{x} \pm 2\,s/\sqrt{n}$. For 90 out of 100, $2\sqrt{0.9 \times 0.1 / 100} = 0.06$, giving 84% to 96%. Halving the number of tests widens the interval by a factor of √2 (about 1.4).

③ How to read a confidence interval correctly

It is tempting to say “the true value lies in this interval with 95% probability”, but strictly that is not quite right. The true value is one fixed number; what changes from test to test is the interval. The 95% describes how often the method of building intervals succeeds: if you built intervals the same way many times, about 95% of them would contain the true value (see the figure above).

In everyday explanations it is fine to say “the true value is probably in this range”. Just remember that the single interval in front of you may be one of the misses — about one time in twenty.

There is one more reading that matters in safety. Safety requirements are usually one-sided, such as “a miss rate of 3% or less”: only being too high is a problem, and being lower does not matter. In that case, instead of the ± 2 × standard error interval, which leaves 2.5% out on each side, you judge by the one-sided upper bound, “estimate + 1.64 × standard error”, which leaves 5% out on the upper side only. Example: 20 misses in 1,000 trials (2%) gives a standard error of about 0.44%; the two-sided 95% interval is 1.1–2.9%, and the one-sided 95% upper bound is about 2.7%. At the same 95% confidence, the upper end of the two-sided interval (2.9%) is stricter than the one-sided bound (2.7%). The “rule of three” in 1.8 is also a one-sided upper bound. When reporting a result, state the confidence level and whether it is one- or two-sided, as in “95% one-sided upper bound 2.7%”. → 5.5

④ Why we divide by n − 1

In 1.4 we divided by the count n when calculating the spread. When estimating the true spread from sample data, however, we divide by n − 1.

$$s^2 = \frac{1}{n-1}\sum_{i=1}^{n}(x_i - \bar{x})^2$$

The reason is that the deviations are measured not from the true mean μ but from the mean computed from the data, $\bar{x}$. Because $\bar{x}$ is, by construction, as close to the data as possible, the deviations come out smaller than they really are. Dividing by n − 1 corrects this underestimate on average. The resulting $s^2$ is called the sample variance, and its square root s the sample standard deviation. For vehicle B in 1.4, dividing by n gives 0.78 m and dividing by n − 1 gives 0.85 m. With plenty of data the difference is tiny, but with only six data points it cannot be ignored.

⑤ When data are scarce, or there are zero failures

With few data points, “2” is not enough. Because the true spread is also being estimated, the uncertainty grows, so we use a value from the t-distribution instead.

Number of data points n	6	11	31	Very many
Value used instead of “2” (95%)	2.57	2.23	2.04	1.96

For vehicle B’s stopping positions in 1.4 (six runs, mean 1.0 m, s = 0.85 m), the 95% confidence interval for the mean is $1.0 \pm 2.57 \times 0.85/\sqrt{6} \approx 1.0 \pm 0.89$ m — so wide that the true mean could plausibly be anywhere from 0.1 m to 1.9 m.

Also, when not a single failure has occurred, the formula above gives $\hat{p} = 0$ with zero width and cannot be used. In that case use the rule of three from 1.8 (the upper bound on the failure rate is about 3/n), which corresponds to the upper end of a one-sided confidence interval.

Summary

A point estimate is a single number; an interval estimate is a range. A 95% confidence interval is roughly “estimate ± 2 × standard error”
The 95% describes how often the method succeeds: built the same way many times, about 95% of intervals contain the true value
Estimate the true spread by dividing by n − 1. With few data use a t-distribution value; with zero failures use the rule of three

Quick quiz

Q1. A detector detected the target in 380 of 400 tests. What are the point estimate and the 95% confidence interval (± 2 × standard error) for the detection rate?

The point estimate is 95%. The standard error is $\sqrt{0.95 \times 0.05 / 400} \approx 0.011$, so the interval is 95% ± 2.2%, roughly 92.8% to 97.2%.

Q2. The requirement is “a detection rate of at least 90%”. The detector succeeded in 93 of 100 tests. Can you say with 95% confidence that the requirement is met?

No. 93% ± $2\sqrt{0.93 \times 0.07 / 100}$ ≈ 93% ± 5.1%, so the lower end is about 87.9%. That is below 90%, so more tests are needed.

Q3. Which describes a 95% confidence interval more accurately? (a) The true value lies in this interval with 95% probability (b) If you built intervals the same way many times, about 95% of them would contain the true value

(b). The true value is fixed; it is the interval that varies. Saying (a) in everyday explanations does little harm, but (b) is what it actually means.

Q4. A vehicle drove 1,000 km with zero accidents. What happens if you build a confidence interval for the accident rate with the ± 2 × standard-error formula?

The estimate is 0 and the width is 0, leading to the false conclusion of “an accident rate of zero”. With zero failures, use the rule of three: the upper bound is about 3 ÷ 1,000 km — you can only say “no worse than about one accident per 333 km”.