A Claim You Can Test

Before anything can be proved it has to be said precisely enough to be wrong. Most disagreements that look mathematical are really two people arguing about an unsharpened claim.

Two people are arguing about whether a change made the system faster. They have

the same measurements in front of them. They are still arguing an hour later.

Nothing is wrong with their reasoning. The problem is that nobody said what

faster means, over what workload, by how much, and the statement they are

arguing about has no truth value to establish.

This lesson is about fixing that, and it comes first because every later

argument in this library assumes it has already been done.

What makes a claim testable

A claim is a statement that is definitely true or definitely false once the

facts are known. Not one that is easy to settle, and not one everybody agrees

about, but one with no third option available.

The test that catches almost everything is a single question: what would I

have to observe to admit that this is false? A statement with a clear answer is

a claim. A statement where every possible observation can be absorbed is not,

however confidently it is held.

FIG 1Four statements, scored on what a sharp claim needs

	subject fixed?	assertion fixed?	range fixed?	threshold fixed?
the model is good	1	0	0	0
the model is accurate on	1	1	0	0
the model scores above 9	1	1	1	1
every request is handled	1	1	1	1
most users prefer the ne	1	1	0	1

The first row fails on three counts and is the shape most disagreements take. The second names a property but leaves hard undefined and says nothing about how accurate. The third and fourth are claims: an observation settles each one and the two parties cannot wriggle afterwards. The last row is the interesting failure, because most is a threshold, more than half, but the range is unfixed: most of which users, measured when. A claim can be precise in three respects and useless in the fourth, and the missing one is usually the range.

The four columns are the four things to fix, and they are worth naming because

sharpening becomes mechanical once you know what you are looking for.

The subject is what the claim is about. The assertion is what property is being

attributed to it. The range is which cases are covered, whether every one of

them, at least one, or some named set. The threshold is the number that

separates holding from failing.

FIG 2From an argument to a claim, in six moves

step	version	what was added	can it be settled?
1	the new version is faster	nothing yet	no, and this is where the hour went
2	the new version serves requests faster	the assertion is now about request time rather than some vague speed	no, still nothing measurable
3	the new version serves requests faster on the checkout path	the subject is narrowed to a specific path	no, faster by what measure
4	the median response time on the checkout path is lower	the measure is named, and median rather than average matters here	nearly, but lower by how much and compared with what
5	the median is at least twenty per cent lower than the current version	a threshold, so a one per cent improvement no longer counts as a win	nearly, but measured on what traffic
6	measured over a full weekday of production traffic, the median checkout response time is at least twenty per cent below the current version	the range of cases is fixed	yes, and both parties know what would settle it

6 steps

Notice what happened between the fourth and fifth rows. Adding the threshold is usually the moment the real disagreement appears, because one person meant any improvement at all and the other meant something worth the risk of deploying. That disagreement was there the whole time and the vague version hid it. This is the main practical value of sharpening a claim: it is not that the precise version is more rigorous, it is that producing it forces the disagreement into the open while it can still be discussed.

Claims you have already written

This is not an exercise confined to arguments. Code is full of claims, and they

are the sharpest kind, because a machine evaluates them.

FIG 3Three kinds of claim, each true or false of the state at that moment

python

def total(order):
    if order.quantity > 0 and order.price >= 0:        # a claim about this order
        amount = order.quantity * order.price
        assert amount >= 0, 'total must never be negative'   # a claim asserted here
        return amount
    raise ValueError('invalid order')

def test_total():
    assert total(Order(quantity=3, price=4)) == 12     # a claim about one case

The condition in the branch is a claim about this particular order, evaluated now. The assertion is a claim that the program declares must be true at that point, and it stops if it is not, which is as sharp as a claim gets. The test case is a claim about one specific input and output. All three are either true or false with no third option, which is why thinking carefully about claims is a practical skill rather than a philosophical one.

The last line is where this course is going. A test case is a claim about one

input. A hundred test cases are a hundred claims about a hundred inputs. None

of them, nor all of them together, is the claim that the function is correct

for every input, and the distance between those two is exactly what the rest of

this course is about.

FIG 4How much of the space a claim covers

Everything to the left of the gap is evidence, and evidence of this kind is valuable and is not the same as a proof: a thousand passing tests are consistent with a function that fails on the thousand and first input. Everything to the right rules out failure for every input at once. Testing is usually the right investment, because proofs are expensive and tests catch the errors people actually make. The reason to know which side of the line you are on is that it determines what you are entitled to say, and the whole of this course is about crossing the gap deliberately rather than pretending it is not there.

Three habits

Before arguing, state the claim in one sentence with its subject, assertion,

range and threshold, and notice which of those you had to invent. The one you

had to invent is where the disagreement is.

Ask what would settle it. If no observation would, stop, because the statement

is not yet something that can be established and no amount of argument will

make it one.

Notice whether your claim covers one case or every case. Both are respectable

and they are not interchangeable, and the next lessons are about moving between

them honestly.

Recap

A claim worth arguing about is one that is definitely true or definitely false once the facts are in. The test is whether you can describe an observation that would settle it, and a claim with no such observation is not yet a claim.
Sharpening a vague statement means fixing four things: what is being talked about, what is being asserted of it, over what range of cases, and against what threshold. Most vague claims are missing the range and the threshold.
Code is full of claims. A condition in an if statement, an assertion, and a test case are each a claim that is true or false of the state at that moment, and the discipline of saying exactly what they assert is the same discipline.

This is the reading half

Starting the course gives you your own copy of it. Every idea on every page has problems standing under it, marked with a reason rather than a tick, and any sentence you do not believe can be opened and argued with. None of that can happen on a page nobody owns.

The contents

NextAnd, Or, and Not →