Why You Must Not Peek

Last timeReading the Result

Checking the result every day and stopping when it looks good turns a one in twenty false alarm rate into something close to one in three, with nobody doing anything dishonest.

The most common way an experiment lies

The experiment is planned for two weeks. On day three somebody looks, and the

treatment is ahead and significant. The team ships it and moves on.

Nobody did anything dishonest. Everybody involved would describe that as

reading the data carefully. And the false alarm rate of that procedure is not

one in twenty.

Why repeated looks inflate the rate

The threshold means one thing: if the change does nothing, a difference this

large appears by chance about one time in twenty. One look, one chance in

twenty.

Look twice and there are two chances. Look fourteen times and the chance that

at least one look crosses the line is more than half.

FIG 1The chance of at least one false win over several looks

: the false alarm rate of a single look, usually one in twenty
: the number of times the result is looked at
: the chance of at least one look showing a false win

The looks are not fully independent, since the data on day four contains the data from day three, which makes the real figure somewhat lower than this. It also makes it worse in another way, which is that a difference that has drifted high tends to stay high for several days, so the stopping rule has many correlated opportunities to fire. The measured answer for daily checking of a fortnight run sits around one in three.

FIG 2False alarm rate against the number of looks

looks treated as independentthe rate everybody believes they have

The flat line is the promise and the curve is the practice. At fourteen looks, which is daily checking of a fortnight, the curve is above a half. The gap between the two lines is the quantity of false findings a peeking team carries, and because the method itself is not at fault, nothing in the output warns anybody that the promise has been broken.

Why early numbers are so unstable

There is a second effect on top of the arithmetic, and it is the one that

makes early stopping so seductive.

The measured difference at any point in a run is the true difference plus a

wobble, and the wobble shrinks like one over the square root of the data so

far. On day two of a fortnight there is a seventh of the data, so the wobble

is about two and a half times larger than it will be at the end.

So early in a run the measured difference wanders, and it wanders through

large values. A team watching that wander and stopping at the first moment it

looks good is not sampling the effect, it is sampling the extremes of the

wobble.

The lesson stops here

2 more paragraphs to go

You have read the opening. The rest of the argument, the problems that check whether it landed, and the lines worth keeping at the end all come with a plan.

The first lesson of every course in the library reads the whole way through, free, so you can see exactly what the rest of them are.

See the plan The contents

This is the reading half

Starting the course gives you your own copy of it. Every idea on every page has problems standing under it, marked with a reason rather than a tick, and any sentence you do not believe can be opened and argued with. None of that can happen on a page nobody owns.

The contents