How Much Traffic It Will Take
Last timeSplitting the Traffic Fairly
The number of people needed goes up as the square of how small a change you want to see, which is why most experiments teams want to run cannot be run at all.
The calculation that ends most discussions
Before running anything, work out how many people it will take. It is one
multiplication and it frequently ends the discussion, which is a service
rather than an obstruction.
- the variability of the measure among individual people
- the size of change you want to be able to detect
- people needed on each side, so twice this in total
Take the checkout example. Conversion runs at four per cent, and the change is
worth having if it lifts conversion by two tenths of a percentage point.
Variability is 0.0384, the change is 0.002, and the calculation gives about
a hundred and fifty-four thousand people on each side.
| step | lift to detect, percentage points | people per side | total people | days at 20,000 visitors a day |
|---|---|---|---|---|
| 1 | 1.0 | 6,100 | 12,200 | 1 |
| 2 | 0.5 | 24,600 | 49,200 | 3 |
| 3 | 0.2 | 153,600 | 307,200 | 16 |
| 4 | 0.1 | 614,400 | 1,228,800 | 62 |
| 5 | 0.03 | 6,826,700 | 13,653,300 | 683 |
Why small effects cost so much
The square in the denominator is the whole story, and it is worth
understanding rather than memorising.
A measured difference between two groups wobbles. The size of the wobble falls
like the square root of the number of people, because that is how averages
behave: four times the people gives half the wobble, not a quarter of it.
To see an effect, the effect has to stand clear of the wobble. So an effect
half as large needs a wobble half as large, which needs four times the people.
An effect a tenth as large needs a hundred times the people.
The lesson stops here
3 more paragraphs to go
You have read the opening. The rest of the argument, the problems that check whether it landed, and the lines worth keeping at the end all come with a plan.
The first lesson of every course in the library reads the whole way through, free, so you can see exactly what the rest of them are.
See the planThe contentsThis is the reading half
Starting the course gives you your own copy of it. Every idea on every page has problems standing under it, marked with a reason rather than a tick, and any sentence you do not believe can be opened and argued with. None of that can happen on a page nobody owns.
The contents