Say the Number, the Range, the Conditions and the Limits
Last timeThe Same Model, Many Different Scores
Everything in this course turns into a short list of things a report has to contain, and a shorter list of claims it is then entitled to make. Neither list is expensive to satisfy.
Everything in the previous seven lessons reduces to a short list. A score is a
fraction of a particular set of questions, it carries sampling error, it moves
under formatting, it inflates when the set is picked over, and it collapses when
the answers were in the training data. The practical consequence is a list of
things to write down, and the list is shorter than most people expect.
- the claim: how much better one system is than the other, on these questions
- both measured on the same question set, so the questions they both get right and both get wrong cancel
- the uncertainty of the difference itself, computed from the disagreements and not from the two scores separately
- two standard errors, which is the conventional width for a range you expect to contain the truth
The minimum
Alongside it go three more facts, none of which takes a sentence: how many
questions, what the conditions were, and how many configurations were tried
before this one was reported. That is the minimum report. Four items, one of
which is a square root of numbers you already have.
| earlier | middle | recent | |
|---|---|---|---|
| how many questions were | 62 | 71 | 78 |
| the extraction and marki | 21 | 29 | 38 |
| an uncertainty range or | 18 | 24 | 31 |
| the exact text sent to t | 9 | 14 | 22 |
| any check for leakage | 4 | 7 | 11 |
Ranks are not results
A leaderboard position is a statement about everyone else. It changes when a
system you have never run is added, when another team retunes, when the test set
is refreshed. A paired difference does not. It is a statement about two systems
on a fixed set of questions, and it stays true when the table around it moves.
The lesson stops here
5 more paragraphs to go
You have read the opening. The rest of the argument, the problems that check whether it landed, and the lines worth keeping at the end all come with a plan.
The first lesson of every course in the library reads the whole way through, free, so you can see exactly what the rest of them are.
See the planThe contentsThis is the reading half
Starting the course gives you your own copy of it. Every idea on every page has problems standing under it, marked with a reason rather than a tick, and any sentence you do not believe can be opened and argued with. None of that can happen on a page nobody owns.
The contents