Knowing a Change Helped

Last timeWhere Things Get Lost

A fixed set of cases, run before and after, is the only thing that distinguishes an improvement from a change. One impressive answer proves nothing at all.

A fixed set of cases

The whole apparatus is one file. Thirty to a hundred cases, each with an input

and a note about what a good answer has to contain, run before a change and

after it.

Three properties make it work. It is fixed, so the two runs are comparable; a

set that grows between runs cannot be compared with itself. It is written down

rather than remembered, so the judgement about what counts as a good answer was

made before anybody saw the output. And it includes the awkward cases from the

second lesson, since a set of ordinary cases cannot detect anything.

FIG 1The whole apparatus

python

CASES = [
  {"input": "where is my order A-4192",
   "must": ["the tracking status", "no invented delivery date"]},
  {"input": "",                                  # empty input
   "must": ["asks what they need"]},
  {"input": "this is the third time I have written",
   "must": ["acknowledges the earlier contacts", "offers a next step"]},
]

def run(version):
    return sum(graded(version, c) for c in CASES) / len(CASES)

before, after = run(old_body), run(new_body)
print(before, after, len(CASES))                 # the three numbers that matter

The three cases shown are a typical one, an empty input and an angry one, which is the shape the set should have throughout. The grading can be done by a person reading the answers, by a program checking for a property, or by a model asked whether each requirement is met, and the three differ in cost and in reliability rather than in principle. What matters more than the grading method is that the same method is applied to both versions.

How many cases you need

The number of cases is not a matter of taste. It is set by the size of the

difference you want to be able to see.

FIG 2Cases needed for a given precision

: the rate you are measuring, as a share
: the wobble you are willing to live with, as a share
: cases needed

A rate near eighty per cent measured to within four points needs a hundred cases. Measured to within two points it needs four hundred, because the requirement is in the denominator squared. This is the arithmetic behind the uncomfortable fact that a twenty case set cannot tell a five point improvement from nothing, and it is why the honest reading of a small set is as a check for large breakages rather than as a measurement.

FIG 3How the wobble shrinks with the number of cases

a rate near 80 per centa rate near 50 per cent

Both curves fall as the square root, which means the first fifty cases buy far more than the next fifty. The practical readings are that thirty cases is enough to catch something badly broken, a hundred is enough to see a change of five points, and several hundred is what a change of two points requires. The upper curve is higher because a rate near one half is the noisiest thing to measure, which is worth knowing when choosing what to measure at all.

The noise in a measurement

With the set in hand, the comparison is two numbers, and the number that

decides whether they differ is a third one nobody computes.

Eighty cases, fifty eight passing before and sixty four after. That is seven

and a half points, which looks like a clear improvement. The wobble on a rate

near three quarters with eighty cases is about five points, and the difference

between two measurements carries the noise of both. Seven and a half points

against that is suggestive and not settled.

The lesson stops here

3 more paragraphs to go

You have read the opening. The rest of the argument, the problems that check whether it landed, and the lines worth keeping at the end all come with a plan.

The first lesson of every course in the library reads the whole way through, free, so you can see exactly what the rest of them are.

See the plan The contents

This is the reading half

Starting the course gives you your own copy of it. Every idea on every page has problems standing under it, marked with a reason rather than a tick, and any sentence you do not believe can be opened and argued with. None of that can happen on a page nobody owns.

The contents