What to Do With the Answer

Last timeWhen a Test Is the Wrong Tool

A measured difference becomes a decision by comparing two costs, the cost of shipping something that does nothing and the cost of discarding something that worked.

The verdict is not the decision

The interval is in. The next question is what to do, and it is not answered by

the statistics, because it depends on what each kind of mistake costs.

There are exactly two ways to be wrong. Ship a change that does nothing, and

pay for maintaining it, the risk it carries, and the complexity it adds.

Discard a change that worked, and give up its gain forever. These two costs

are almost never similar, and the decision rule follows from their ratio.

FIG 1Four changes, and what each mistake costs

	cost of shipping a dud,	cost of discarding a win	ratio
wording on one button	1	400	400
a redesigned checkout	120	400	3
a new ranking model	300	900	3
a change to how refunds	2000	300	1

Rough yearly figures in thousands. The first row is the common case teams handle badly: a copy change costs almost nothing to keep and the lost gain is large, so the bar for shipping should be very low and is usually set at the same one in twenty as everything else. The last row is the opposite: a refund change that goes wrong is expensive and hard to unwind, so it deserves a much higher bar than the convention, and the convention does not supply one.

FIG 2The chance of being right at which shipping becomes the better bet

: the gain if the change works
: the loss if it does not
: the chance the change works at which shipping and not shipping are equally good

A gain of four hundred thousand against a loss of a hundred and twenty gives a break even chance of twenty three per cent, so a change with a one in four chance of working is worth shipping. This is the arithmetic the conventional threshold replaces with a single number for every decision, and it is worth doing by hand for anything consequential: the answer is sometimes ship on weak evidence and sometimes do not ship on strong evidence, and the convention gives neither.

The two mistakes, priced

Three decision patterns come out of this.

For a cheap, reversible change with a large potential gain, ship on weak

evidence. The interval straddling zero is not a reason to discard it if the

cost of being wrong is a day of work and a revert.

For an expensive or hard to reverse change, demand much more than the

convention, and in particular demand that the whole interval sit above the

threshold rather than merely excluding zero.

And for anything in between, which is most things, the rule from the fourth

lesson applies: the interval against the threshold, with unresolved as a

permitted answer.

FIG 3The break even chance, against how much the gain outweighs the loss

break even chancea coin flip

Where the gain is ten times the loss, a one in eleven chance justifies shipping. Where they are equal, it takes a coin flip. Where the loss is five times the gain, it takes an eighty three per cent chance. The curve is steep on the left, which is why cheap experiments on cheap changes are worth running even when they will probably fail: the asymmetry does the work, not the evidence.

Most experiments fail

The uncomfortable finding from every large programme that has published its

numbers is that most changes do not work. Depending on the organisation and

the maturity of the product, somewhere between a third and a tenth of changes

produce the improvement their authors expected.

That is not a criticism of the people proposing them. It is what it looks like

to work on something where the obvious improvements have already been made,

and it has two consequences.

The first is that a programme of experiments should be judged on what it

discards, not on its hit rate. The value is in the changes that were not

shipped, each of which would otherwise have added permanent complexity for

nothing.

The second is that the prior matters when reading a single result. If one

change in ten works, then a bare significant result at the usual threshold is

closer to even odds than to a certainty, because the false wins from the nine

failures are comparable in number to the true win from the one success. This

is why a surprising win is worth replicating and an expected win usually is

not.

The lesson stops here

2 more paragraphs to go

You have read the opening. The rest of the argument, the problems that check whether it landed, and the lines worth keeping at the end all come with a plan.

The first lesson of every course in the library reads the whole way through, free, so you can see exactly what the rest of them are.

See the plan The contents

This is the reading half

Starting the course gives you your own copy of it. Every idea on every page has problems standing under it, marked with a reason rather than a tick, and any sentence you do not believe can be opened and argued with. None of that can happen on a page nobody owns.

The contents