Answers · The mathematics underneath
How do you make a vague claim precise enough to tell whether it's true?
In shortFix what is being talked about, what is asserted, which cases are covered and the cut-off, then name an observation that would prove it false.
Make the claim precise by fixing four slots, the subject, the assertion, the range of cases and the threshold, then name one observation that would settle it. If no observation could settle it, the statement is not yet a claim, however confidently it is held.
Take 'the new version is faster'. It has no measure, no range and no cut-off, so the same measurements can support both sides. Narrow the subject to requests on the checkout path, make the assertion about median response time, fix the range to a full weekday of production traffic, and set the threshold at twenty per cent below the current version. The result is: measured over a full weekday of production traffic, the median checkout response time of the new version is at least twenty per cent below the current version. That sentence can be checked.
A new median of 450 ms fails, because 450 is above 400. A new median of 390 ms passes. The threshold is not decoration. It is what turns 'faster' into a statement with a truth value, and it exposes the real disagreement: one person may have meant any improvement, the other meant an improvement worth deploying.
| Slot | Question it answers | Filled in for the checkout claim |
|---|---|---|
| Subject | What is being talked about? | checkout path under production traffic |
| Assertion | What property is being attributed? | median response time |
| Range | Which cases are covered? | a full weekday of production traffic |
| Threshold | What number separates holding from failing? | at least twenty per cent below the current version |
The natural guess that fails
The natural guess is to repeat the vague sentence more carefully, replacing 'faster' with 'significantly faster' or 'much faster'. That does not help. Each adverb is another threshold with no number and no range, so the claim can absorb every observation: faster on checkout but slower on search, faster on the mean but slower on the median, faster on Monday but slower on Friday. If every possible measurement leaves both parties able to say they were right, no measurement can settle the argument. The test is not whether the sentence sounds precise. The test is whether you can describe an observation that would make you admit it is false.
Producing the precise version has a practical value beyond rigour: it forces the hidden disagreement into the open while it can still be discussed. If one person meant any improvement and the other meant an improvement worth the risk of deployment, the vague sentence hid that disagreement for an hour. The threshold makes it explicit. That is why adding the number often feels like the argument has changed subject. It has not. It has found the subject.
The range is not always every case. A claim can be about one input, a named set, or every input. 'This request is handled within 200 milliseconds' is a claim about one request. 'Every request is handled within 200 milliseconds' is a claim about all requests. The words every and some are part of the claim, not decoration. If you leave the range out, two people can agree on the assertion and the threshold and still be arguing about different sets.
Code makes the same demand in a sharper form, because a machine evaluates the claim. In 'if order.quantity > 0 and order.price >= 0', the condition is a claim about this order at this moment: either both inequalities hold or they do not. The assertion 'amount >= 0' is a claim that the program declares must be true at that point and stops if it is not. The test 'total(Order(quantity=3, price=4)) == 12' is a claim about one input, and the arithmetic is 3 × 4 = 12. All three have no third option.
One case or every case
A test case is a claim about one input. Many test cases are many claims about many inputs. None of them, nor all of them together, is the claim that the function is correct for every input. Passing tests are consistent with a function that fails on an input nobody tried. Testing is evidence, and evidence can only fail to find a problem. An argument over every case is proof, and proof rules the problem out for every input at once. Both are respectable, and they are not interchangeable.
Where sharpening stops
A claim can be precise in three slots and useless in the fourth, and the missing one is usually the range. 'Most users prefer the new layout' has a threshold, more than half, but it does not say which users or when they were measured. Change the range and the truth value can change. If an assumption changes, restate the claim: a threshold of twenty per cent on one weekday is not the same claim as twenty per cent on every weekday. If no observation would settle the claim even after all four slots are filled, stop: it is not something that can be established.
Worth keeping
- A claim is testable only when you can describe an observation that would make you admit it is false.
- Sharpening a claim fixes its subject, assertion, range and threshold, and the missing one is usually the range.
- A passing test is evidence about one input, not a proof about every input.
Where this goes next
This question is one step of Arguing That It Always Holds, a course that works through its subject from the beginning. The page that teaches this part of it is A Claim You Can Test, and it reads the whole way through whether or not you have an account.
What an account changes is the practice: problems standing under every idea, marked with a reason rather than a tick, in a copy of the course that is yours.
Read that pageThe whole courseOther questions people ask about this
- Why does the negation of "A and B" become "not A or not B"?Negating 'A and B' asserts that at least one of the two fails, so the and becomes an or.
- Why does induction need a base case if the step works?The step only says that if one case holds then the next does; without a first case there is nothing to start from, so no case is proved.
- Why is an if-then statement true when the condition is false?An if-then claim only forbids one case, the condition holding while the result fails, so when the condition is false there is nothing left for it to forbid and it stays true.
- What is the difference between variance and standard deviation?They measure the same spread: variance is the average squared distance from the mean, standard deviation is its square root, so you report the standard deviation and calculate with the variance.
- How do you find the parabola that best fits a function near a point?Match the function's height, slope and bending at the chosen point; those three conditions single out exactly one parabola.
- Why does a test for a rare disease give so many false positives if the test is accurate?A positive result means the test fired, not that you are likely to be ill; when the disease is rare, healthy people vastly outnumber ill ones, so a small false-alarm rate on them swamps the few true cases.