Answers · Putting a model into service

Why bother with a confidence interval when the result is already significant?

In shortA significant result only rules out zero; the interval shows how large the effect might be and whether it is worth acting on.

Because significance answers only whether zero is a plausible explanation. The interval answers the questions you actually have to act on: how large the effect might be, whether it is worth having, and what sizes the experiment has ruled out. A result can be significant and still be consistent with an effect too small to pay for itself, or with an effect that ranges from nothing to comfortably worth having.

The interval around the measured difference

Take conversion of 4.00 per cent on one side and 4.14 per cent on the other, so the measured difference is 0.14 percentage points. Write the base rate as $p = 0.04$ . The variance of a proportion is $p(1-p)$ , which is $0.04 \times 0.96 = 0.0384$ . With $n = 153{,}600$ people on each side, the variance of the difference is the sum of the two side variances: $0.0384/153{,}600 + 0.0384/153{,}600$ . Each term is $0.00000025$ , or $2.5 \times 10^{-7}$ , so the sum is $5 \times 10^{-7}$ . The standard error is the square root, about $0.000707$ , which is $0.0707$ percentage points. Twice that is $0.1414$ percentage points. The interval is therefore $0.14 \pm 0.14$ percentage points, or $0.00$ to $0.28$ percentage points.

\hat{\delta} \;\pm\; 2\sqrt{\frac{\sigma_A^2}{n_A} + \frac{\sigma_B^2}{n_B}}

That interval is exactly the awkward case the verdict hides. The result is significant, because the interval starts at zero, but it also runs to 0.28 points. If the threshold you care about lies inside that range, the interval straddles it: the experiment has not told you whether the change clears the bar. The significance statement and the practical statement are different sentences, and only the interval carries the second. The width was fixed before the run, by traffic and variability, so it could have been predicted during planning rather than discovered at the end.

What the verdict does not say

The natural guess is that significant means large, certain, or worth shipping, and that a 95 per cent interval means a 95 per cent chance the true value lies inside this particular interval. Both guesses fail. Significance is computed by assuming the change did nothing, then asking how often a difference at least this large would appear by chance. If that tail probability is below one in twenty, the result is called significant. The claim is only that a difference this large would be unusual under no effect.

It does not say the chance the change does nothing is one in twenty. That would be a probability about the hypothesis, and the data alone do not supply it; it needs a prior view of how plausible the change was before the run. It does not say the effect is large, because enough traffic makes a tiny lift significant. And it does not say the effect is worth having, which compares the interval with a threshold chosen from costs and consequences.

The interval has a similarly exact reading. Before the data arrive, the procedure makes intervals that catch the true difference 95 per cent of the time across repeated experiments. After you have computed one interval, the true difference is fixed; this interval either contains it or it does not. The 95 per cent belongs to the method, not to this one interval as a posterior probability.

Where the interval changes the decision

The reading rule is to put the interval next to the threshold that matters. If the whole interval is above the threshold, ship. If the whole interval is below it, do not ship, and record that the question is settled for that threshold. If the interval straddles it, the experiment has not answered the question: run longer, decide on other grounds, or say plainly that it is unresolved.

The same rule fixes the null result. An interval that excludes every worthwhile effect makes a null result decisive: the change does nothing worth having. An interval so wide that it excludes almost nothing means the run was too small to tell. Both get written up as no difference, and only the interval separates a decisive null from an underpowered one.

The last case is the one that trips up teams with a lot of traffic. A lift can be real, significant, and narrow in interval, and still be far below the threshold that would pay for the change. Significance is not a size test. The question is never whether the difference is exactly zero; for almost any change it is not. The question is whether the interval sits wholly above the threshold you wrote down before the run. If it sits wholly below, the answer is no, even when the verdict says significant.

Worth keeping

A significant result does not tell you the effect is large or worth having.
An interval around the measured difference says what sizes the experiment has ruled out.
A 95 per cent interval is a procedure that catches the true value 95 per cent of the time across repeats, not a 95 per cent chance for this one interval.

Where this goes next

This question is one step of Running an Online Experiment, a course that works through its subject from the beginning. The page that teaches this part of it is Reading the Result, and it reads the whole way through whether or not you have an account.

What an account changes is the practice: problems standing under every idea, marked with a reason rather than a tick, in a copy of the course that is yours.

Read that page The whole course