Answers · Putting a model into service
Why do the numbers look good while users keep complaining?
In shortBecause the score measures a fixed set of cases with a fixed rule, and the users are living through behaviour that the set and the rule never covered.
Because the two numbers are measuring different populations, and the offline score was never a measurement of what users do. An offline score is a fixed set of cases marked by a fixed rule: it rises whenever the cases it happens to contain improve, whether or not those cases still resemble the traffic arriving today, and whether or not the rule rewards what people actually want.
Take a plain case. A thousand evaluation cases sit in three groups. The change you shipped helps two of them a lot and hurts the third badly, and the third is a category that barely existed when the set was assembled.
| group | share of set | share of traffic | old version | new version |
|---|---|---|---|---|
| search-style questions | 70% | 60% | 0.80 | 0.86 |
| editing requests | 25% | 18% | 0.78 | 0.79 |
| follow-up questions | 5% | 22% | 0.75 | 0.55 |
The new version scores 0.827 against 0.7925, up three and a half points, and both of the large groups improved. Read the contributions one at a time and the arithmetic is honest about what it is doing. The first group gained 0.06 across 70 per cent of the set, worth , or 4.2 points. The second gained 0.01 across 25 per cent, worth 0.25 of a point. The third fell 0.20 across 5 per cent, costing , one point. Add them: points. The group that got much worse was outvoted.
The set is not the traffic
Now weight the same scores by what people actually send rather than by what the set contains. Follow-up questions are 22 per cent of traffic and were 5 per cent of the set.
The sign flips. On the set the release is up 3.45 points; on the traffic it is down 0.62, from 0.7854 to 0.7792. Nothing was mismeasured in either calculation. The set-weighted average is a correct answer to the question the set asks, and the set had stopped asking the question the users were answering. That is why the numbers look good: they are good, about a world that is 5 per cent follow-up questions.
The natural response is to go looking for a bug, and the bug hunt fails for a structural reason rather than a lack of care. A bug is a hypothesis about behaviour the metric reads, and it predicts a fall where the metric is looking. If the breakdown shows every group up or flat, no such bug can exist, and the harm is arriving through a channel the per-answer score does not connect to: latency on the first word, an answer formatted as a long list that reads badly on a phone, memory that stops tracking two turns back, a refusal rate that climbed because the system got more careful. None of those appear in a score that reads one answer at a time with no clock running. Searching harder in the place the evidence has already excluded is what the week after a release usually looks like.
So run the cheap tests in order of cost. Break the aggregate down by group first, an hour's work, because it rules out the sharpest cause or names it outright. Then classify a week of traffic into the set's own groups and compare proportions, which is how the 5 per cent against 22 per cent becomes visible. Then mark fifty unhappy cases by hand with the current rubric. If the rubric approves of answers users disliked, the rule is the thing that must move, and that is a different repair from adding cases to a set that simply never contained them.
None of this applies when the difference is inside the error bars. A couple of points on a thousand cases can be noise, and the first job then is more cases rather than a diagnosis. It also stops applying when the sampled unhappy cases come back marked and the rubric disapproves of them: that is evidence for the boring explanation, that the complaints are unrepresentative, and it should be said plainly rather than argued around. What remains true either way is that the score is a measurement of whatever it was built to measure, and a gap between it and the users is information about that choice before it is information about the users.
Worth keeping
- An offline score can rise on every case it contains while the product gets worse, if the cases no longer resemble the traffic.
- A bug in the behaviour a metric reads would show up as a fall in that metric, so a rising metric points at behaviour the metric does not read.
- Breaking an aggregate down by group costs about an hour and either names the cause or rules out the sharpest one.
Where this goes next
This question is one step of Judging a System, Not a Model, a course that works through its subject from the beginning. The page that teaches this part of it is When the Score Rises and Users Complain, and it reads the whole way through whether or not you have an account.
What an account changes is the practice: problems standing under every idea, marked with a reason rather than a tick, in a copy of the course that is yours.
Read that pageThe whole courseOther questions people ask about this
- My score went up two points — is that a real improvement or just noise?No. At four hundred cases a two point gain is inside the noise, so treat it as no result.
- Should my evaluation cases be real user queries or ones I write myself?Use real user queries, sampled by behaviour rather than at random, plus a held-out part nobody tunes on. Cases you invent measure what you imagined users ask.
- How many cases do I need before the evaluation can even see a small improvement?About four hundred cases per set can see only a six-point move; a two-point improvement needs roughly four thousand two hundred per set.
- How small can my test set be and still catch a regression before a user does?About sixty auto-checked cases, run in two minutes on every change, blocking only on outright failures such as a case that used to pass now failing, never on a fall in the overall score.
- Why does the model return JSON wrapped in extra text when I asked for JSON?A request for JSON does not force the model to return only JSON, so you need to validate the reply and repair it once rather than assume it will parse.
- How do I make the model answer from the documents I give it instead of its own memory?Fence the documents as labelled reference material, put them before the question, tell the model to prefer them over memory, and require a passage number for each claim.