Answers · Putting a model into service

Why does detecting a small improvement take so many more users?

In shortBecause the noise you are trying to see past shrinks only with the square root of the sample, so halving the change you want to spot takes four times the people.

The traffic needed grows with the square of how small a change you are chasing, so halving the lift you want to detect multiplies the people required by four, not by two. Written out, $n = 16\sigma^2/\delta^2$ people on each side, and the $\delta$ in the denominator is squared.

n \;=\; \frac{16\,\sigma^2}{\delta^2}

Take a checkout converting at four per cent. For a yes or no outcome the variability is the rate times one minus it, so $0.04 \times 0.96 = 0.0384$ , which costs nothing to work out in advance. Suppose the change is worth having only if it lifts conversion by two tenths of a percentage point, so $\delta = 0.002$ . Then $16 \times 0.0384 = 0.6144$ and $\delta^2 = 0.000004$ , giving $0.6144 / 0.000004 = 153{,}600$ people per side, or 307,200 in total. At 20,000 visitors a day that is 15.4 days, rounded up to whole weeks, so 16 days.

FIG 1

lift to detect, percentage points	people per side	total people	days at 20,000 a day
1.0	6,144	12,288	1
0.5	24,576	49,152	3
0.2	153,600	307,200	16
0.1	614,400	1,228,800	62

Why the square, and not a straight ratio

The natural guess is that a change half as big needs twice the traffic, because almost everything else about an experiment scales that way. It fails because what the effect has to stand clear of is not the total quantity of noise in the data but the wobble in the measured difference between the two groups, and that wobble shrinks like the square root of the number of people. Four times the people gives half the wobble, not a quarter. So an effect half as large needs a wobble half as large, and a wobble half as large costs four times the people.

n\!\left(\tfrac{\delta}{2}\right) \;=\; \frac{16\,\sigma^2}{(\delta/2)^2} \;=\; 4 \cdot \frac{16\,\sigma^2}{\delta^2} \;=\; 4n(\delta)

The same statement read backwards is the half people forget. Quadrupling the traffic buys a factor of two in resolution, not a factor of four; merely doubling it buys about 1.41, since $\sqrt{2} = 1.41$ . That is why the step from two tenths of a point to one tenth costs 614,400 per side instead of 307,200, and why the jump from one per cent to a quarter of a per cent is a sixteenfold increase rather than a fourfold one. The curve of people against target is nearly flat on the right and nearly vertical on the left, and discussions about what effect size to aim for belong in front of that curve rather than in the abstract, because a concession that sounds small is where the budget dies.

FIG 2People needed against the lift you want to detect

four per cent baseline conversion

Four per cent baseline throughout: thousands per side, with in percentage points. Moving the target from one point to half a point looks like a small concession and costs four times the traffic.

What can be moved instead

Of the three quantities in the sum, only one is comfortably negotiable. The confidence is conventional: relaxing the false alarm rate from one in twenty to one in ten saves roughly twenty per cent of the traffic and costs exactly what it sounds like. The effect size is the decision itself, so lowering it to make the arithmetic work means deciding to detect changes too small to act on. Variability is the real lever, and measuring conversion rather than revenue per person is the biggest single win available, worth something like seventy per cent of the traffic, because a yes or no outcome is far steadier than a sum dominated by a handful of large purchases. Adjusting for each person's earlier behaviour before the run is worth around forty per cent, at the price of needing history on everybody, and trimming the most extreme values is worth about twenty five per cent provided the trimming rule is fixed in advance.

Two things change the answer rather than refine it. Running the test across the whole site instead of the pages the change can affect does not gather more data; it adds people who could not possibly be affected and dilutes the difference, raising the traffic needed rather than lowering it. And there is a ceiling: past about four weeks, people clear cookies and get reassigned, the site changes underneath the test, and the season moves, so a calculation that lands on three months is not a plan but a finding that the question will not be settled this way. The sixteen is itself a planning constant, $(1.96 + 0.84)^2 = 7.84$ doubled for two groups and rounded up, which assumes the normal approximation holds. For very rare events, or samples in the low hundreds, that approximation is shaky and the number it produces is a floor rather than a promise.

Worth keeping

The people needed on each side is sixteen times the variability divided by the square of the change you want to detect.
Halving the change you want to detect multiplies the traffic by four, and reducing it to a quarter multiplies it by sixteen.
Most proposed experiments need more traffic than the site has, and finding that out before the run turns a wasted fortnight into a decision about what to do instead.

Where this goes next

This question is one step of Running an Online Experiment, a course that works through its subject from the beginning. The page that teaches this part of it is How Much Traffic It Will Take, and it reads the whole way through whether or not you have an account.

What an account changes is the practice: problems standing under every idea, marked with a reason rather than a tick, in a copy of the course that is yours.

Read that page The whole course

Why does detecting a small improvement take so many more users?

Why the square, and not a straight ratio

What can be moved instead

Worth keeping

Where this goes next

Read alongside