When a Test Is the Wrong Tool
Last timeTesting Many Things at Once
Three situations defeat an ordinary split test: users who affect each other, effects that take months to appear, and outcomes too rare to measure in any run you can afford.
Three situations where the apparatus fails
Everything so far has assumed three things quietly. That one person
experiencing the change does not affect anybody else. That the effect appears
within the run. And that the outcome happens often enough to count.
Each of these fails in real situations, and the failure is not visible in the
output: the experiment produces a clean interval and a confident verdict,
which is wrong. This lesson is about recognising the three cases before
running anything.
When the groups affect each other
A delivery marketplace tests a change that makes couriers accept jobs faster.
Treated couriers accept faster, take more jobs, and earn more. The experiment
reports a large win.
The jobs the treated couriers took were jobs the control couriers would have
taken. The control group did worse because the treatment group existed, so the
measured difference is the real effect plus the amount the treatment took from
the control, which can be most of it. Ship the change to everybody and the
gain largely disappears, because there is nobody left to take from.
| what the treated user co | direction of the error | unit that removes it | |
|---|---|---|---|
| a two sided marketplace | 1 | 1 | 1 |
| a social feed change | 2 | 1 | 2 |
| a shared inventory | 1 | 1 | 1 |
| a shared queue or suppor | 3 | 1 | 3 |
| a change to prices in on | 4 | 2 | 1 |
The repair is to randomise something large enough to contain the interference.
Whole cities, whole markets, clusters of socially connected people, or whole
time periods alternating between the two versions. The arithmetic of the third
lesson still applies, with the number of units now being the number of cities
rather than the number of people, which usually means an effect has to be
several times larger before it can be detected.
Time periods are the most practical version for a marketplace. Run the old
version for half an hour, the new one for the next half hour, and alternate
for weeks. Every period sees the whole market, so there is no interference,
and the comparison is between periods rather than between people. The cost is
that periods differ from one another in ways people do not, so a large number
of them is needed.
The lesson stops here
6 more paragraphs to go
You have read the opening. The rest of the argument, the problems that check whether it landed, and the lines worth keeping at the end all come with a plan.
The first lesson of every course in the library reads the whole way through, free, so you can see exactly what the rest of them are.
See the planThe contentsThis is the reading half
Starting the course gives you your own copy of it. Every idea on every page has problems standing under it, marked with a reason rather than a tick, and any sentence you do not believe can be opened and argued with. None of that can happen on a page nobody owns.
The contents