1.7 Hypothesis testing

Build intuition

A perception model that detects pedestrians has been updated. From a long track record of tests, the old version is known to have a detection rate of 90%. The new version detected the pedestrian in 94 out of 100 tests. Can we say it has “improved by 4 points”?

As we saw in 1.5 and 1.6, test results vary. Even if the true detection rate were still 90%, 100 tests could produce 94 detections by chance. The question is whether this result can be dismissed as luck.

So hypothesis testing deliberately assumes that “the new version has not changed at all (it is still 90%)”, and then calculates how likely a result like ours would be. If it would be very unlikely, we conclude that the assumption was wrong and judge that “there is a difference”. If it could easily happen, we cannot claim a difference.

It works like the presumption of innocence in a courtroom: start from “not guilty” (no difference) and return “guilty” (a difference) only when the evidence is strong enough to overturn it.

With 94 out of 100, even if nothing had changed, the probability of a result “4 or more points away from 90%” is about 18%. Something that happens about one time in five by chance is no basis for saying “it improved”. With the same 94% from 376 out of 400, however, that probability drops to about 0.8%. That is hard to explain by chance, so we judge that there is a difference. The same “94%” supports different conclusions depending on the number of tests — just as with confidence intervals in 1.6.

Check with math

① Null and alternative hypotheses

The assumption of “no difference” is called the null hypothesis \(H_0\), and “there is a difference” the alternative hypothesis \(H_1\) (see Notation). The null hypothesis is the one we set up hoping to reject. Writing the old version’s detection rate as \(p_0 = 0.90\):

Here a change in either direction — better or worse — counts as a “difference” (a two-sided test).

② How many standard errors away: z

Using the standard error under the null hypothesis (1.5) as the ruler, we measure how many of them separate the observed value \(\hat{p}\) from the assumed value \(p_0\).

For 94 out of 100, \(z = (0.94 - 0.90)/\sqrt{0.9 \times 0.1/100} = 0.04/0.03 \approx 1.33\); for 376 out of 400, \(z = 0.04/0.015 \approx 2.67\). Comparing means (such as stopping positions) works the same way, with \(s/\sqrt{n}\) in the denominator. With few data, use a t-distribution value as in 1.6 (a t-test).

③ p-values and the significance level

The p-value is the probability, assuming the null hypothesis is true, of a result at least as extreme as the one observed. Using the normal distribution from 1.3, it follows from z:

If the p-value is below a threshold fixed in advance — the significance level α, usually 5% — we reject the null hypothesis and say there is a “statistically significant difference”. This is the same as |z| exceeding about 2 (1.96, to be precise). In our example, 100 tests (p ≈ 18%) are not significant, while 400 tests (p ≈ 0.8%) are. Note that the table uses the normal approximation; with few tests, it differs slightly from an exact binomial calculation.

Be careful: a p-value is not the probability that there is no difference. It says “if there were no difference, how unusual would this result be?” To reason about the probability that a hypothesis itself is true, you need the Bayesian approach from 1.2.

④ The link to confidence intervals

The assumed value (90%) falling outside the 95% confidence interval (1.6) corresponds, roughly, to significance at the 5% level.

• 94 of 100: 94% ± 4.7% → 89.3% to 98.7% (contains 90% → not significant)
• 376 of 400: 94% ± 2.4% → 91.6% to 96.4% (excludes 90% → significant)

A test gives a yes-or-no answer — “there is a difference” or “we cannot say there is” — whereas a confidence interval also tells you how large the difference is. When reporting results, include the interval, not just the p-value.

If the old version’s 90% was itself measured in tests, its error adds to the uncertainty. The standard error of the difference combines the two as \(\sqrt{\mathrm{SE}_1^2 + \mathrm{SE}_2^2}\). If the old version had also scored 360 out of 400 (90%), z would drop from 2.67 to about 2.1 (p ≈ 4%) — only just significant.

⑤ Two kinds of error, and power

A test’s verdict can be wrong in two different ways.

A 5% significance level is a promise to keep type I errors to 5%. How often type II errors occur, on the other hand, depends heavily on the number of tests. If the new version really had improved to 94%, the probability of judging “there is a difference” at the 5% level (the power) would be:

With 100 tests, a real improvement still ends in “no significant difference” more than 70% of the time. “No significant difference” does not prove “no difference”. When planning tests, it is common to decide first on the smallest difference you do not want to miss, then choose a number of tests that gives a power of at least 80%.

Summary

• Hypothesis testing assumes “no difference” (the null hypothesis) and checks whether the result can be explained by chance
• z = (observed value − assumed value) ÷ standard error. If the p-value is below the significance level (usually 5%) — that is, if |z| exceeds about 2 — the difference is “significant”. This roughly corresponds to the 95% confidence interval excluding the assumed value
• No significant difference ≠ no difference (few tests mean low power). Significant ≠ large. Many comparisons produce chance “significant” results

Quick quiz

Further reading

• Next lesson: 1.8 Statistics of rare events
• Related: 1.2 Conditional probability and Bayes’ theorem, 1.5 Law of large numbers and the central limit theorem, 1.6 Estimation and confidence intervals
• Glossary: Hypothesis testing, p-value / significance level, Type I error / type II error / power, Multiple comparisons
• Video lecture: YouTube (about 5 minutes) (the key points explained with slides and narration; the Japanese article has a Japanese video)