Basis Lab

Answers · Putting a model into service

My score went up two points — is that a real improvement or just noise?

In shortNo. At four hundred cases a two point gain is inside the noise, so treat it as no result.

Probably not. At four hundred cases a two point gain is inside the uncertainty of the measurement, and a set that size cannot honestly detect a difference smaller than about six and a half points between two independent runs.

The arithmetic starts with one score. A score is a proportion measured on a sample, so repeating it on a fresh sample moves it. The standard error is s=p(1−p)/ns=\sqrt{p(1-p)/n}, with pp the score as a fraction and nn the number of cases. At p=0.70p=0.70 and n=400n=400, p(1−p)=0.21p(1-p)=0.21, so s=0.21/400=0.000525=0.0229s=\sqrt{0.21/400}=\sqrt{0.000525}=0.0229. That is 2.3 percentage points.

s=p(1−p)ns = \sqrt{\frac{p(1-p)}{n}}

Take a baseline of 71 and a new score of 73. The baseline has a one-standard-error band of roughly 71 plus or minus 2.3, or 68.7 to 73.3. The new score has a band of 73 plus or minus 2.3, or 70.7 to 75.3. Those bands overlap heavily. A frozen system resampled at this size will often move two or three points without anything having changed.

The difference is the number that matters

Comparing two scores is not comparing one score with zero. Each score carries its own error, and subtraction does not cancel the errors; they combine in quadrature. For two independent sets of the same size, sdiff=s2+s2=s2s_{\text{diff}}=\sqrt{s^2+s^2}=s\sqrt{2}. With s=2.3s=2.3 points, sdiff=2.3×1.414=3.25s_{\text{diff}}=2.3\times1.414=3.25, about 3.3 points. A difference worth claiming should be roughly twice its error, so about 2×3.3=6.62\times3.3=6.6, which the table rounds to 6.5. Your two points is below even the 3.3 point error on the difference.

sdiff=s12+s22=2s≈3.3 points at n=400s_{\text{diff}} = \sqrt{s_1^2 + s_2^2} = \sqrt{2}s \approx 3.3\text{ points at } n=400

The natural guess is that two points is two points, and the only question is whether the sign is positive. That would be right if the score were a census of every case the system will ever see. It is a sample. The correct comparison is not 2 against 0 but 2 against the difference's error, and at 400 cases that error is 3.3 points. A two point move is what a frozen system does on a different draw of cases. If both scores come from the same 400 cases and you compare per case, the paired design helps: the smallest detectable difference falls to about 55/400=2.7555/\sqrt{400}=2.75 points, under three. Two points is still under that line.

FIG 1
cases in each seterror on one scoreerror on the differencesmallest difference worth claiming
1004.66.513.0
4002.33.36.5
10001.52.14.1
40000.71.02.1

What to put on the dashboard

Compute the smallest detectable difference once and write it above the numbers. At 400 cases it is about 6.5 points for two independent runs, or about 2.75 points if the same cases are paired. The sentence becomes: this evaluation can detect differences of 6.5 points or more; below that, report no result. That sounds like a weakness, but the alternative is a team keeping four point moves that are noise, and roughly half of the changes kept that way make things worse. The threshold is a property of the set size, not of the result.

The answer changes with nn. At 1000 cases the error on one score is about 1.5 points, the error on the difference about 2.1, and the smallest difference worth claiming about 4.1. At 4000 cases those become about 0.7, 1.0 and 2.1. So a two point gain on 4000 cases is still below the clean threshold, though only just. At 100 cases the smallest difference worth claiming is about 13 points, and a two point move is pure noise. Substitute your own nn before treating any two point move as real.

One more trap is the dashboard itself. If twenty metrics each have a one in twenty chance of looking like a finding when nothing changed, you expect about one apparent finding per look: 20×(1/20)=120\times(1/20)=1. A two point move in one subgroup is exactly the number the eye finds. Decide in advance which number is under test, and require a surprise from any secondary number to be confirmed on fresh cases before anybody acts on it.

Worth keeping

  • At four hundred cases the standard error on a score near 70 per cent is about 2.3 points, and on the difference between two independent scores about 3.3 points.
  • A four-hundred-case set cannot honestly detect a difference smaller than about six and a half points between independent runs.
  • Two points is below the error on the difference, so it is not evidence of improvement.

Where this goes next

This question is one step of Judging a System, Not a Model, a course that works through its subject from the beginning. The page that teaches this part of it is Is That a Real Improvement, and it reads the whole way through whether or not you have an account.

What an account changes is the practice: problems standing under every idea, marked with a reason rather than a tick, in a copy of the course that is yours.

Read that pageThe whole course

Other questions people ask about this

  1. Why do the numbers look good while users keep complaining?Because the score measures a fixed set of cases with a fixed rule, and the users are living through behaviour that the set and the rule never covered.
  2. Should my evaluation cases be real user queries or ones I write myself?Use real user queries, sampled by behaviour rather than at random, plus a held-out part nobody tunes on. Cases you invent measure what you imagined users ask.
  3. How many cases do I need before the evaluation can even see a small improvement?About four hundred cases per set can see only a six-point move; a two-point improvement needs roughly four thousand two hundred per set.
  4. How small can my test set be and still catch a regression before a user does?About sixty auto-checked cases, run in two minutes on every change, blocking only on outright failures such as a case that used to pass now failing, never on a fall in the overall score.
  5. Why does the model return JSON wrapped in extra text when I asked for JSON?A request for JSON does not force the model to return only JSON, so you need to validate the reply and repair it once rather than assume it will parse.
  6. How do I make the model answer from the documents I give it instead of its own memory?Fence the documents as labelled reference material, put them before the question, tell the model to prefer them over memory, and require a passage number for each claim.

Read alongside