Reading the Result
Last timeHow Much Traffic It Will Take
Report the measured difference with an interval around it, because the interval says what the experiment ruled out and the usual yes or no verdict does not.
Report a range, not a verdict
The experiment has ended. Conversion was 4.00 per cent on one side and 4.14
per cent on the other, so the measured difference is 0.14 percentage points.
That number is not the effect. It is one noisy measurement of the effect, and
the useful report says how noisy. The standard way to say it is an interval:
the measured difference plus or minus about twice the standard error.
- the measured difference between the two sides
- the variability of the measure on each side
- the number of people on each side, which is units and not events
The interval is the honest report because it carries two things at once. It
says what was measured, and it says what the experiment can rule out. The
verdict carries only the first and discards the second.
| step | measured lift | interval | the usual verdict | what the interval says |
|---|---|---|---|---|
| 1 | +0.42 | +0.28 to +0.56 | significant | a real effect, comfortably worth having |
| 2 | +0.14 | +0.00 to +0.28 | significant, just | anywhere from nothing to worth having |
| 3 | +0.11 | -0.03 to +0.25 | not significant | anywhere from slightly harmful to worth having |
| 4 | +0.01 | -0.02 to +0.04 | not significant | any real effect is too small to matter |
What significance actually asserts
The usual threshold works like this. Assume the change did nothing at all.
Compute how often, under that assumption, a difference at least as large as
the one measured would appear by chance. If that is below one in twenty, call
the result significant.
So the claim is: a difference this large would be unusual if the change did
nothing. That is all it is, and it is worth listing what it does not say.
It does not say the chance the change does nothing is one in twenty. That is a
different quantity, and computing it needs something the data does not
contain, namely how plausible the change was before the experiment. For a team
whose changes work one time in ten, a bare significant result is closer to a
coin flip than to a certainty.
It does not say the effect is large. With enough traffic, a lift of one
thousandth of a percentage point is significant.
And it does not say the effect is worth having, which is a question about
costs and lives in the first lesson and the last.
The lesson stops here
3 more paragraphs to go
You have read the opening. The rest of the argument, the problems that check whether it landed, and the lines worth keeping at the end all come with a plan.
The first lesson of every course in the library reads the whole way through, free, so you can see exactly what the rest of them are.
See the planThe contentsThis is the reading half
Starting the course gives you your own copy of it. Every idea on every page has problems standing under it, marked with a reason rather than a tick, and any sentence you do not believe can be opened and argued with. None of that can happen on a page nobody owns.
The contents