Answers · Putting a model into service
What does a p-value actually tell me?
In shortA p-value is the chance of seeing a difference at least as large as yours if the change did nothing; it is not the chance the result was luck or that there is no effect.
A p-value is the probability of seeing a difference at least as large as the one you measured, if the change did nothing at all. It is not the probability that your result was luck, and it is not the probability that there is no effect.
Take the conversion test with 153,600 people on each side and a four per cent base rate. The variance on each side is . Dividing by 153,600 gives , and adding the two sides gives . The square root is about , which is percentage points, so twice the standard error is about percentage points. The measured difference was points: per cent minus per cent. That is two standard errors from zero. Under the assumption that the change did nothing, a difference at least that large in either direction appears with probability about . That probability is the p-value. It is a statement about the data under a fixed assumption, not a statement about the assumption given the data.
| measured lift (points) | interval (points) | usual verdict | what the interval says |
|---|---|---|---|
| +0.42 | +0.28 to +0.56 | significant | a real effect, comfortably worth having |
| +0.14 | 0.00 to +0.28 | significant, just | anywhere from nothing to worth having |
| +0.11 | -0.03 to +0.25 | not significant | anywhere from slightly harmful to worth having |
| +0.01 | -0.02 to +0.04 | not significant | any real effect is too small to matter |
The interval carries everything the verdict carries and more. Its width was fixed before the experiment ran: it depends on the traffic and the variability, both known in advance. That is why the width can and should be predicted during planning. The verdict discards the width and keeps only whether the interval crosses zero.
What it does not say
The natural reading is that a p-value of 0.05 means a 5 per cent chance the result was luck, or a 5 per cent chance the change does nothing. That is a different quantity: the chance the change does nothing given the data. Computing it needs the plausibility of the change before the experiment, which the data does not contain. For a team whose changes work one time in ten, a bare significant result is closer to a coin flip than to a certainty. The p-value answers a question about the data, not about the hypothesis.
Nor does a small p-value say the effect is large or worth having. With enough traffic a lift of one thousandth of a percentage point is significant. The question is never whether a difference exists, since for almost any change it is not exactly zero; the question is whether the difference is big enough to act on. In the table above the word significant is the same in all three verdict rows and is doing no work in any of them.
Where it stops
Not significant does not mean no effect. An interval from minus 0.02 to plus 0.04 points has excluded every effect worth having, and the conclusion that the change does nothing worth having is correct and valuable. An interval from minus 0.9 to plus 1.1 points has excluded almost nothing, and the only correct conclusion is that the experiment was too small to tell. Both get written up as no difference, and only the interval distinguishes them. A p-value alone cannot tell you which situation you are in.
The reading procedure is to put the interval next to the threshold written down before the run. If the whole interval is above the threshold, ship. If the whole interval is below it, do not ship, and record that the question is settled. If the interval straddles it, the experiment has not answered the question: run longer, decide on other grounds, or say honestly that it is unresolved. An experiment that excluded every worthwhile effect has answered the question. An experiment that excluded nothing has not run long enough, and saying so is a better outcome than recording a false conclusion.
Worth keeping
- A p-value is the probability of a difference at least as large as the observed one, assuming the change did nothing.
- A p-value of 0.05 does not mean a 5 per cent chance of no effect; that would require the prior plausibility of the change.
- An interval around the measured difference says what the experiment ruled out, which a yes or no verdict does not.
Where this goes next
This question is one step of Running an Online Experiment, a course that works through its subject from the beginning. The page that teaches this part of it is Reading the Result, and it reads the whole way through whether or not you have an account.
What an account changes is the practice: problems standing under every idea, marked with a reason rather than a tick, in a copy of the course that is yours.
Read that pageThe whole courseOther questions people ask about this
- Why bother with a confidence interval when the result is already significant?A significant result only rules out zero; the interval shows how large the effect might be and whether it is worth acting on.
- How do I decide whether a small but significant win is actually worth shipping?Ship only if the chance the gain is real beats the cost of being wrong divided by that cost plus the gain. That bar is rarely five per cent.
- Why can't I A/B test a change where users can affect each other?A split assumes one person's result does not depend on what version others got. When users compete for the same jobs or stock, the control group is changed by the treatment, so the comparison is biased.
- Why does detecting a small improvement take so many more users?Because the noise you are trying to see past shrinks only with the square root of the sample, so halving the change you want to spot takes four times the people.
- Why does the model return JSON wrapped in extra text when I asked for JSON?A request for JSON does not force the model to return only JSON, so you need to validate the reply and repair it once rather than assume it will parse.
- Why do the numbers look good while users keep complaining?Because the score measures a fixed set of cases with a fixed rule, and the users are living through behaviour that the set and the rule never covered.