Saying What Would Count as Better

One measure and one size of change worth acting on, both fixed in writing before any data arrives, because a measure chosen afterwards can always be found that shows a win.

The question that gets skipped

A team changes the checkout page. Two weeks later the numbers are in and

somebody asks whether it worked.

That question cannot be answered, because nobody said what working would look

like. There are forty measures on the dashboard, three of them are up, two are

down, and the rest have not moved. Whichever story the team tells now, the

data will support it.

Everything in this course that goes wrong goes wrong here, before any data was

collected. This lesson is about the two decisions that have to be made first.

One measure, named first

Pick one measure. Name it before the experiment starts. Everything else is a

guardrail, watched to make sure nothing broke, not evidence that the change

worked.

The reason is arithmetic. The usual threshold accepts a result as a win when

something that extreme would happen by chance at most one time in twenty.

Watch twenty measures that have not actually changed and the chance that at

least one of them crosses that line is about two thirds.

FIG 1The chance of at least one false win, by how many measures are watched

nothing has actually changed

At one measure the chance is the five per cent everybody expects. At fourteen it is above a half, so a team watching fourteen unchanged measures is more likely than not to find a winner. At forty it is 87 per cent. Nothing in this curve requires anybody to behave dishonestly: it is what happens when a dashboard is read after a change.

The same arithmetic applies to the variations nobody counts as measures.

Looking at the result for new users separately, and for mobile separately, and

for one country separately, is looking at more measures. So is trying the

comparison with outliers removed, and then with a different window. Each

variant is another chance for noise to cross the line, and the count that

matters is the number of comparisons anybody could have made, not the number

they report.

How much better is worth having

The second decision is how large a change would be worth acting on, and it is

the one teams skip entirely.

It is not a matter of taste. A change costs something to build, to review and

to maintain. It earns something per user. Below some size the change is not

worth having even if it is real, and that size can be computed before anything

is run.

FIG 2The smallest change worth acting on

: the yearly cost of building and maintaining the change
: how many people pass through the affected place in a year
: what one additional conversion is worth
: the increase in conversion rate at which the change pays for itself

The arithmetic is deliberately crude and still decisive. Forty thousand pounds a year of cost, two million people through the page, sixty pounds per conversion gives a break even lift of about three hundredths of one per cent. Anything above that is worth shipping if it is real, which is a much lower bar than most teams assume and changes what the experiment has to be able to detect.

The number matters because it feeds directly into the next two lessons. A

change worth acting on at three hundredths of a per cent needs an enormous

amount of traffic to detect, and finding that out before the run rather than

after is the difference between an experiment and two wasted weeks.

FIG 3Four changes, and the size at which each becomes worth having

	yearly cost in thousands	people through, in milli	value per conversion	break even lift, hundred
a copy change on one but	2	2	60	2
a redesigned checkout fl	120	2	60	100
a new recommendation mod	400	8	15	333
a feature needing a supp	900	2	60	750

Lifts in hundredths of a per cent, so 100 means one per cent. The copy change pays for itself at almost any real effect, which is why such changes are worth testing cheaply and shipping on weak evidence. The feature needing a support team has to move conversion by seven and a half per cent to be worth having, which is an enormous effect, and the useful conclusion is to be reached before the work is done rather than after.

Proxies, and what they miss

The thing anybody actually wants is rarely measurable this week. Satisfaction,

trust, whether somebody is still a customer in two years: all real, none

available at the end of a two week run.

So experiments measure proxies. Clicks instead of interest. Sessions instead

of value. Immediate purchases instead of lifetime purchases. This is necessary

and it introduces the most common serious failure in the subject, which is a

change that improves the proxy by damaging the thing it stands for.

FIG 4Four proxies and the change that beats each one

	available within two wee	a change that improves i
clicks on the recommenda	1	1
time spent in the app	1	2
immediate conversion	1	3
sign ups	1	4

The second column codes the defeating change: 1 is a misleading thumbnail, which lifts clicks and lowers satisfaction, 2 is a harder to use navigation, which lifts time spent because people are lost, 3 is a hidden cancellation policy, which lifts purchases and raises refunds, 4 is removing the price from the sign up page, which lifts sign ups and fills the funnel with people who will not pay. Every one of these ships as a win if the proxy is the only thing watched.

Two habits keep this honest. Name what the proxy misses at the moment it is

chosen, in a sentence, and pick a guardrail measure that would move if the

proxy were being gamed. And wherever the real outcome is measurable eventually,

measure it eventually, on a sample, so the relationship between the proxy and

the outcome is checked at least occasionally rather than assumed forever.

Writing it down first

Four things get fixed in writing before the run starts: the measure, the size

worth acting on, how long the run will take, and the rule for what will be

done with each possible result.

Each one of these can be adjusted after the fact to turn a null result into a

win. A different measure, a smaller threshold, a few more days, a decision

rule invented once the number is known. None of these adjustments feels like

cheating from inside, which is exactly why the protection has to be mechanical

rather than ethical.

FIG 5What gets written down before the run

python

plan = {
    "measure": "completed checkouts per visitor",
    "worth_acting_on": 0.0003,        # cost over traffic times value per conversion
    "guardrails": ["refund rate", "support contacts per order", "page load time"],
    "run_until": "2026-11-14",        # a date, computed from the traffic needed
    "decision": {
        "interval_above_threshold": "ship",
        "interval_straddles_zero": "do not ship, and record the interval",
        "any_guardrail_worse": "do not ship whatever the measure says",
    },
}
print("four decisions, all made while the answer is still unknown")

Nothing here is elaborate and all of it is unavailable afterwards. The decision block is the part most often missing, and it is the one that stops a result from being argued about: write down now what you will do if the answer is no, because the version of you that writes it then is the only one with no stake in the outcome.

Written down, it also becomes checkable by somebody else, which is the second

purpose. An experiment nobody could audit is a claim rather than a

measurement.

The next lesson takes the plan as given and deals with the mechanism

underneath it, which is splitting the traffic in a way that makes the two

groups genuinely comparable.

Recap

Choosing the measure after seeing the data guarantees a win, because with enough measures one of them has moved in the right direction by chance.
The size of change worth acting on comes from arithmetic about what the change costs and what it earns, not from what the experiment turns out to be able to detect.
Most measures anybody can collect quickly are proxies for the thing actually wanted, and naming what the proxy misses is part of choosing it.

This is the reading half

Starting the course gives you your own copy of it. Every idea on every page has problems standing under it, marked with a reason rather than a tick, and any sentence you do not believe can be opened and argued with. None of that can happen on a page nobody owns.

The contents

NextSplitting the Traffic Fairly →