Answers · Putting a model into service
How small can my test set be and still catch a regression before a user does?
In shortAbout sixty auto-checked cases, run in two minutes on every change, blocking only on outright failures such as a case that used to pass now failing, never on a fall in the overall score.
Sixty cases, taking about two minutes, run on every change and allowed to stop the merge. Not a faster version of the full evaluation: a separate, deliberately narrower set, drawn from things that have already broken in front of a user, each with an answer a script can check without a judge. Everything larger runs behind it and reports without blocking anybody.
The count comes from the clock, not from wanting more coverage. Two minutes is 120 seconds, and 120 seconds across sixty cases is two seconds a case. Forty to eighty cases fit that slot; a hundred and fifty do not. Once the number is fixed by the time budget, the question stops being how much to measure and becomes what such a small set is still entitled to say.
That is 6.5 percentage points on a score, one standard error, so the 95% interval is about points either side. A gate that blocks on a two-point fall is watching a quantity it cannot see at this size: the fall is a third of one standard error, a of , and an unchanged system produces a drop that large about 38% of the time. Nearly four runs in ten would stop a merge that changed nothing.
| tier | cases | minutes per run | runs per week | blocks a merge |
|---|---|---|---|---|
| fast set | 60 | 2 | 300 | yes |
| nightly full set | 600 | 55 | 1 | no |
| weekly paired comparison | 400 | 180 | 0.2 | no |
What the gate blocks on
The natural guess has two halves and both fail. The first is that a small set is the big set at lower resolution, so protection scales with size and you should run the thorough evaluation on everything. It does not scale that way. The fast set is not measuring the same claim more coarsely; it is checking a different claim, that nothing that used to work has stopped working, and that claim is cheap. The second half is putting the score in the gate: block if it drops more than two points. On sixty cases a two-point fall is indistinguishable from the rise or fall you get by sampling different cases, and the arithmetic above puts the false-alarm rate near forty per cent. A threshold does not become safe by being small. It becomes safe by being above the noise, and on this many cases nothing useful is.
Block on absolutes instead: a case that passed on the previous commit and fails now, a crash or a timeout, an empty or truncated answer, a safety check failing, output that is supposed to be a particular shape and is not. Each is a yes or no about a named case, so the failure message says which case and what it did, and the person who caused it fixes it in minutes rather than opening an investigation. None of these needs a threshold or an interval, which is precisely why they survive at a sample size this small: they are not estimates.
The delay is what the size buys. With cost written as for hours to detection, one hour gives , while twenty-four hours gives , roughly forty-five times as much. The straight part is users meeting the fault; the curved part is commits landing on top of it and an author reloading context they no longer have. Against that, two minutes is not a compromise on rigour. It is the only tier close enough to the change to be believed.
Where it stops holding
Two things break the arrangement, and neither is mathematical. The first is a check that fails without a cause. It keeps running after quarantine and keeps reporting, but it leaves the gate the same day, because one intermittent failure teaches the re-run habit, and the re-run habit disables the fifty good checks standing next to it. Pin the seed, pin the wording, and if the cause is still unexplained, it does not block. The second is coverage. A set grown from incidents only guards the kinds of failure that have already happened once. A new failure mode, a group that got worse while the total held, a tone that is wrong without being checkable, is invisible to sixty closed-answer cases, and that is the job of the nightly judged run and the weekly paired comparison. A case whose correct answer depends on phrasing cannot join the fast set at all, because it will flip on wording rather than on behaviour.
Worth keeping
- On sixty cases the standard error of a score is about 6.5 percentage points, so a gate that trips on a two-point drop is measuring noise rather than a regression.
- A regression caught in two minutes rather than a day costs roughly forty-five times less once you count the commits landing on top of it and the context the author has to reload.
- A blocking check that fails once without a cause should leave the gate the same day, because one flaky check teaches the re-run habit that disables every check beside it.
Where this goes next
This question is one step of Judging a System, Not a Model, a course that works through its subject from the beginning. The page that teaches this part of it is Catching It Before a User Does, and it reads the whole way through whether or not you have an account.
What an account changes is the practice: problems standing under every idea, marked with a reason rather than a tick, in a copy of the course that is yours.
Read that pageThe whole courseOther questions people ask about this
- Why do the numbers look good while users keep complaining?Because the score measures a fixed set of cases with a fixed rule, and the users are living through behaviour that the set and the rule never covered.
- My score went up two points — is that a real improvement or just noise?No. At four hundred cases a two point gain is inside the noise, so treat it as no result.
- Should my evaluation cases be real user queries or ones I write myself?Use real user queries, sampled by behaviour rather than at random, plus a held-out part nobody tunes on. Cases you invent measure what you imagined users ask.
- How many cases do I need before the evaluation can even see a small improvement?About four hundred cases per set can see only a six-point move; a two-point improvement needs roughly four thousand two hundred per set.
- Why does the model return JSON wrapped in extra text when I asked for JSON?A request for JSON does not force the model to return only JSON, so you need to validate the reply and repair it once rather than assume it will parse.
- How do I make the model answer from the documents I give it instead of its own memory?Fence the documents as labelled reference material, put them before the question, tell the model to prefer them over memory, and require a passage number for each claim.