Testing Many Things at Once
Last timeWhy You Must Not Peek
Watching twenty measures or running five variants multiplies the chances for noise to cross the line, so a win on the eleventh measure is more likely to be noise than a finding.
The same arithmetic, pointed sideways
The previous lesson was one measure looked at many times. This one is one look
at many measures, and the arithmetic is identical.
Twenty measures on the dashboard, none of them actually changed. The chance
that at least one of them crosses the usual threshold is about two thirds. The
team finds that one, writes it up, and has a finding.
- the threshold applied to each measure
- how many measures, variants or slices were examined
- the chance that at least one of them crosses
| how many, typically | counted by the team | counted by the arithmeti | |
|---|---|---|---|
| measures on the dashboar | 20 | 1 | 20 |
| variants against one con | 4 | 1 | 4 |
| slices of the audience | 12 | 0 | 12 |
| versions of the analysis | 3 | 0 | 3 |
| weeks the run could have | 3 | 0 | 3 |
Correcting for the family
The simplest repair is to divide the allowance. If twenty measures are being
examined and the overall false alarm rate should stay at one in twenty, then
each measure gets one four hundredth.
- the false alarm rate wanted across the whole set
- the number of measures being examined
- the threshold each individual measure has to cross
A less conservative alternative controls the share of the reported findings
that are false, rather than the chance of any false finding at all. Instead of
one flat tight threshold, the measures are sorted by how extreme they are and
the threshold is tightened progressively down the list. When a team is
genuinely screening many measures to see which are worth following up, this is
the better tool, because it keeps the share of nonsense among the findings
under control without demanding that every single one be beyond doubt.
The lesson stops here
2 more paragraphs to go
You have read the opening. The rest of the argument, the problems that check whether it landed, and the lines worth keeping at the end all come with a plan.
The first lesson of every course in the library reads the whole way through, free, so you can see exactly what the rest of them are.
See the planThe contentsThis is the reading half
Starting the course gives you your own copy of it. Every idea on every page has problems standing under it, marked with a reason rather than a tick, and any sentence you do not believe can be opened and argued with. None of that can happen on a page nobody owns.
The contents