The Name on the Box Is a Claim, Not a Description
Last timeWhen the Answers Were in the Training Data
A benchmark called reasoning measures whatever its questions happen to reward. Shortcuts, wrong answer keys and narrow coverage all open a gap between the name and the measurement.
A benchmark is called a test of reasoning. It contains four hundred questions.
The name is not a description of those questions; it is somebody's claim that
answering them requires reasoning. That claim can be wrong, and when it is
wrong, every number ever reported on the set inherits the error.
The shortcut test
The sharpest check on the second box costs almost nothing. Delete part of each
question and see how well a system does on what remains. If the score stays well
above chance, the part you deleted was not necessary, and whatever is left is
enough to answer.
| score | chance | above chance | |
|---|---|---|---|
| full question and passag | 88 | 25 | 63 |
| passage removed | 71 | 25 | 46 |
| question removed, option | 62 | 25 | 37 |
| content words scrambled | 29 | 25 | 4 |
This is not a hypothetical failure mode. It has been found repeatedly, in
reading comprehension sets where the answer could be guessed from the passage
alone, in inference sets where the hypothesis alone was enough, and in multiple
choice sets where the correct option was reliably the longest. None of these
were discovered by the people who built the sets, because nobody thinks to run a
system on half a question.
The reason shortcuts appear so reliably is worth understanding, because it is
structural rather than careless. Questions are produced quickly and at volume,
usually by many people working to a template, and the correct answer and the
wrong ones are almost never produced the same way. The correct one often comes
from a source document and the distractors are invented. That difference leaves
a trace in length, in phrasing, in how specific the wording is, and a system
with no understanding of the subject can learn the trace. The remedy is equally
structural: generate the distractors by the same process as the answer, or have
a second person write them without being told which is which.
The lesson stops here
6 more paragraphs to go
You have read the opening. The rest of the argument, the problems that check whether it landed, and the lines worth keeping at the end all come with a plan.
The first lesson of every course in the library reads the whole way through, free, so you can see exactly what the rest of them are.
See the planThe contentsThis is the reading half
Starting the course gives you your own copy of it. Every idea on every page has problems standing under it, marked with a reason rather than a tick, and any sentence you do not believe can be opened and argued with. None of that can happen on a page nobody owns.
The contents