Answers · Putting a model into service
Does a benchmark score tell you how good a model really is, or just how it did on those questions?
In shortNo: a benchmark score is an average over chosen questions and a chosen marking rule, so it describes that test more than the model in general.
No. A benchmark score tells you how a particular model did on a particular sample of questions under a particular marking rule, and it becomes an estimate of general quality only if that sample and rule resemble the use you care about. The published percentage is an average of ones and zeros, not a measured property of the model.
What the arithmetic says
Suppose a set has 400 questions and the model is marked right on 336 of them. Then , or 84 per cent. That is the whole calculation: 0.84 times 400 is 336, and 336 divided by 400 is 0.84. If those 400 were a random draw from a larger population, the standard error of the average is . With and , ; dividing by 400 gives ; the square root is . One standard error is therefore about 1.8 percentage points. A rough 95 per cent interval is , which is 0.804 to 0.876, or 80.4 to 87.6 per cent. Even under the friendliest assumption, the number is an estimate with a visible width.
Why the natural guess fails
The natural guess is that 84 per cent is the model's accuracy in general. That would require the 400 questions to be a random sample of all questions you might ask, and the marking rule to match your own notion of correct. Neither is usually true. The questions were chosen by somebody before any model was involved. Take a hundred-question set made of 38 recall questions, 29 one short step from a stated fact, 21 passage lookups and 12 arithmetic items: those sum to 100, and 38+29=67, so two thirds of the set is recall or one short step. The set could still be published under a single subject name. A model that memorises well and reasons poorly would do well there, and the name would not warn you. The score is accurate about that sample; the claim that it measures the model in general is a separate claim about how the sample was collected.
The marking rule can move the score as much as the questions. Take identical responses to 100 questions and four defensible rules. Exact match marks 61 right, substring match marks 74, a judge model marks 79, and a human reader marks 77. The spread between the strictest and the most generous is points. None of those rules is wrong. Exact match is reproducible and punishes a right answer phrased differently; substring match is forgiving and sometimes accepts a right string inside a wrong sentence; a judge model brings its own preferences; a human reader is the reference but too slow for scale. A score quoted without the rule is a length quoted without units.
| n | right | s | standard error | rough 95 per cent interval |
|---|---|---|---|---|
| 100 | 84 | 0.84 | 0.0367 | 0.768 to 0.912 |
| 400 | 336 | 0.84 | 0.0183 | 0.804 to 0.876 |
| 1600 | 1344 | 0.84 | 0.00917 | 0.822 to 0.858 |
When it stops being true
The interval in the table is conditional on two assumptions: the questions are a random draw from a defined population, and the marking rule is fixed and matches the thing you care about. If the questions were scraped from exam papers, the population is exam papers, not your colleagues' questions. Running more exam questions narrows the interval around exam performance; it cannot tell you whether exam performance transfers. If the marking rule is exact match, the score is about exact match, not about a reader's judgement. Change the instruction that comes with each question, or allow the model to think before answering, and the same model can move. The score belongs to the model, the question set and the conditions together. Quoting it as a fact about the model alone drops two of the three.
What to ask instead
When you see a score, ask: a sample of what, marked how, run under what conditions. If any of those answers is missing, you have a number without the information needed to interpret it. The appropriate response is not scepticism about the model but scepticism about the claim. A vendor can choose the question set and the marking rule that flatter a system without anybody lying. The number is real; the authority often attached to it is not.
Worth keeping
- A benchmark score is a sample average over chosen questions marked by a chosen rule, so it belongs to the model, the set and the conditions together.
- Sampling error around a score can be calculated, but it does not cover the error from choosing a non-representative set or a mismatched marking rule.
- A score can be accurate on its own questions and still fail to transfer to the questions you actually care about.
Where this goes next
This question is one step of Reading a Benchmark Honestly, a course that works through its subject from the beginning. The page that teaches this part of it is A Score Is a Sample, and it reads the whole way through whether or not you have an account.
What an account changes is the practice: problems standing under every idea, marked with a reason rather than a tick, in a copy of the course that is yours.
Read that pageThe whole courseOther questions people ask about this
- Why does the model return JSON wrapped in extra text when I asked for JSON?A request for JSON does not force the model to return only JSON, so you need to validate the reply and repair it once rather than assume it will parse.
- Why do the numbers look good while users keep complaining?Because the score measures a fixed set of cases with a fixed rule, and the users are living through behaviour that the set and the rule never covered.
- How do I make the model answer from the documents I give it instead of its own memory?Fence the documents as labelled reference material, put them before the question, tell the model to prefer them over memory, and require a passage number for each claim.
- Why does the memory used by a model keep growing as the conversation gets longer?The model's weights do not grow. It keeps a cache of keys and values for every earlier token, so memory rises with conversation length and with how many conversations run at once.
- How many legitimate requests should I be willing to block in order to catch the bad ones?Block no more legitimate requests than one prevented harmful answer is worth: the break-even count is the harm cost divided by the refusal cost.
- What does a p-value actually tell me?A p-value is the chance of seeing a difference at least as large as yours if the change did nothing; it is not the chance the result was luck or that there is no effect.