Basis Lab

Putting a model into service

Decide whether a change actually improved anything, by designing the split, working out in advance how much traffic the answer needs, reading the result honestly, and recognising the handful of ways an experiment reports a win that is not there

Running an Online Experiment

The A/B test, from the arithmetic of how long to run to the discipline of not peeking. Mostly a course about how experiments go wrong.

8 lessons, written and corrected before you arrived. Reading them here needs no account. The first reads the whole way through; the others open and then stop, because a page nobody owns cannot tell who is reading it. Starting the course gives you your own copy, where every idea has problems standing under it and you can ask about any sentence.

Start reading

This is course 8 of 9 in a reading track: put a model in front of people and know whether it is working. The track is an order, not a gate, and the rest of it is on one page.

  1. 01Saying What Would Count as BetterOne measure and one size of change worth acting on, both fixed in writing before any data arrives, because a measure chosen afterwards can always be found that shows a win.
  2. 02Splitting the Traffic Fairlyopening onlyThe two groups have to differ only in the change being tested, which means assigning by a stable hash of the right unit and checking afterwards that the split came out as intended.
  3. 03How Much Traffic It Will Takeopening onlyThe number of people needed goes up as the square of how small a change you want to see, which is why most experiments teams want to run cannot be run at all.
  4. 04Reading the Resultopening onlyReport the measured difference with an interval around it, because the interval says what the experiment ruled out and the usual yes or no verdict does not.
  5. 05Why You Must Not Peekopening onlyChecking the result every day and stopping when it looks good turns a one in twenty false alarm rate into something close to one in three, with nobody doing anything dishonest.
  6. 06Testing Many Things at Onceopening onlyWatching twenty measures or running five variants multiplies the chances for noise to cross the line, so a win on the eleventh measure is more likely to be noise than a finding.
  7. 07When a Test Is the Wrong Toolopening onlyThree situations defeat an ordinary split test: users who affect each other, effects that take months to appear, and outcomes too rare to measure in any run you can afford.
  8. 08What to Do With the Answeropening onlyA measured difference becomes a decision by comparing two costs, the cost of shipping something that does nothing and the cost of discarding something that worked.