Splitting the Traffic Fairly

Last timeSaying What Would Count as Better

The two groups have to differ only in the change being tested, which means assigning by a stable hash of the right unit and checking afterwards that the split came out as intended.

The one thing a split has to achieve

Two groups, and the only difference between them is the change. Everything

else, who they are, when they arrived, what device they are on, what the

weather was, has to be the same on average.

Randomness is what buys this. Nobody can list all the ways two groups of

people might differ, so nothing careful or deliberate can equalise them.

Flipping a coin for each person equalises all of them at once, including the

ones nobody thought of, because the coin knows nothing about the person.

That is the entire theoretical content of this lesson. The rest is the

mechanics, which is where it goes wrong.

Which unit gets randomised

Assign the person, not the request. If a visitor sees the new checkout on one

page and the old one on the next, they have experienced neither version and

their behaviour tells you about a muddle.

The rule is that the unit has to be large enough that one unit never meets

both versions. For most interface changes that is the person. For a change

affecting people who interact, such as a shared document or a two sided

marketplace, the unit may have to be a whole group, a city or a time period,

which the seventh lesson deals with.

FIG 1The number that matters is the number of units

: the number of events, such as sessions or page views
: events per unit, such as sessions per person
: the number of independent units, which is what the arithmetic uses

Four hundred thousand sessions from eighty thousand people with five sessions each is eighty thousand units, not four hundred thousand. Treating the sessions as independent overstates the amount of evidence fivefold and produces intervals that are far too narrow, which is the single most common way a confident result turns out to be noise. The next lesson computes with n, and this is where n comes from.

Assigning by hash

The mechanism is a hash. Take the person stable identifier, join it to the

name of the experiment, hash the result to a number, and take that number

modulo a hundred. Below fifty is one side, fifty and above is the other.

Each ingredient is there for a reason. The stable identifier means the same

person gets the same side on every visit, across devices if they are signed in

and for the whole run. Hashing means nothing has to be stored, so there is no

table to grow, to go stale or to be unavailable when the page is being

rendered. And the experiment name in the hash means two experiments running at

once assign independently, instead of putting the same unlucky people in every

treatment group.

The lesson stops here

3 more paragraphs to go

You have read the opening. The rest of the argument, the problems that check whether it landed, and the lines worth keeping at the end all come with a plan.

The first lesson of every course in the library reads the whole way through, free, so you can see exactly what the rest of them are.

See the plan The contents

This is the reading half

Starting the course gives you your own copy of it. Every idea on every page has problems standing under it, marked with a reason rather than a tick, and any sentence you do not believe can be opened and argued with. None of that can happen on a page nobody owns.

The contents