Answers · Putting a model into service
Why can't I A/B test a change where users can affect each other?
In shortA split assumes one person's result does not depend on what version others got. When users compete for the same jobs or stock, the control group is changed by the treatment, so the comparison is biased.
You can run the experiment, but you cannot read a user-level split as the effect of shipping the change. A 50/50 split by user is unbiased only when one user's outcome depends on that user's version and not on which versions other users received. A marketplace, a social feed and a shared stock room all break that condition: the control group is exposed to the treatment through the people it competes or interacts with.
Why the control stops being a control
Take 100 couriers split 50/50 and 100 jobs. In the old version each group accepts 50 jobs. In the test the treated couriers accept 60 and the control accept 40. The reported difference is 20 jobs per 50 couriers. Only 10 jobs moved: the treated group took 10 that the control would have taken, and the control lost 10. With equal baselines the measured difference is the true effect plus twice the diverted amount, so 20 = 0 + 2*10. Ship it to everyone and there is no control left to take from; total accepted jobs is still 100, so the 20 disappears.
The natural guess is that randomisation is a property of the split alone: assign half the users to A, half to B, and the difference is causal. Randomisation is only causal when the assignment of one user does not change another user's outcome. If a control courier loses a job to a treated courier, the control outcome is not the outcome that control would have had under no treatment. The bias is usually toward a win, because the treatment takes from the control. Shared prices can go either way: a treated seller cutting price may lower the control's price or raise it, so the sign is not knowable after the fact.
Shared inventory is the same arithmetic in a different costume. If the treatment group buys the last item, the control group sees a sold-out page. The control conversion rate falls for a reason that has nothing to do with the control experience. A social feed is the same: a treated user posts more, a control user sees more posts from treated friends, so the control is partly treated. The unit that removes it is a cluster of connected users or a whole market, not a single user.
Randomise a bigger container
The repair is to randomise a unit large enough to contain the interference: whole cities or markets for a marketplace, clusters of connected people for a social feed, whole time periods for a switchable market. Then the comparison is between independent containers, not between users who can affect each other. The price is precision. The number of independent units falls from hundreds of thousands of people to tens of cities. If an ordinary split with 400,000 people can detect 0.20 percentage points, a city split with 40 cities can detect 1.50 points. That is 1.50/0.20 = 7.5 times larger. Alternating half-hour periods with 600 periods detects 0.90 points, 0.90/0.20 = 4.5 times larger. The arithmetic is the same as any sample-size calculation, but n is now the number of cities or periods.
Time periods are the most practical version for a marketplace. Run the old version for half an hour, the new one for the next half hour, and alternate for weeks. Every period sees the whole market, so there is no interference, and the comparison is between periods rather than between people. The cost is that periods differ from one another in ways people do not, so a large number of them is needed.
You cannot repair this by checking covariates or by running longer. A longer run estimates the same contaminated difference more precisely. You cannot subtract the spillover because you never observe the control group without the treatment group in the same market. The only repairs are design repairs: change the randomisation unit, change the time structure, or change the question.
When it is still fine
None of this says a user-level test is always wrong. If the treated share is small, the resource is not scarce, and users do not interact, the interference is negligible and the ordinary split is fine. If interference is strong, an ordinary split is not a weak experiment; it is measuring a different quantity, the effect of being treated in a world where others are not. That quantity is not what shipping decides. A cluster design measures the effect of shipping to a whole market, which is the decision, but it needs an effect several times larger to be seen. If the effect is also slow or rare, cluster randomisation does not repair that; it has to be combined with a holdback or with an earlier measure.
Worth keeping
- An ordinary split test estimates a causal effect only when one user's outcome does not depend on which versions other users received.
- When users share jobs, stock, attention or prices, the control group is changed by the treatment group and the measured difference is not the effect of shipping the change.
- The repair is to randomise whole markets, clusters or time periods, which reduces the number of independent units and therefore requires a much larger effect to detect.
Where this goes next
This question is one step of Running an Online Experiment, a course that works through its subject from the beginning. The page that teaches this part of it is When a Test Is the Wrong Tool, and it reads the whole way through whether or not you have an account.
What an account changes is the practice: problems standing under every idea, marked with a reason rather than a tick, in a copy of the course that is yours.
Read that pageThe whole courseOther questions people ask about this
- What does a p-value actually tell me?A p-value is the chance of seeing a difference at least as large as yours if the change did nothing; it is not the chance the result was luck or that there is no effect.
- Why bother with a confidence interval when the result is already significant?A significant result only rules out zero; the interval shows how large the effect might be and whether it is worth acting on.
- How do I decide whether a small but significant win is actually worth shipping?Ship only if the chance the gain is real beats the cost of being wrong divided by that cost plus the gain. That bar is rarely five per cent.
- Why does detecting a small improvement take so many more users?Because the noise you are trying to see past shrinks only with the square root of the sample, so halving the change you want to spot takes four times the people.
- Why does the model return JSON wrapped in extra text when I asked for JSON?A request for JSON does not force the model to return only JSON, so you need to validate the reply and repair it once rather than assume it will parse.
- Why do the numbers look good while users keep complaining?Because the score measures a fixed set of cases with a fixed rule, and the users are living through behaviour that the set and the rule never covered.