Answers · How models work
When should a model refuse to answer instead of guessing?
In shortRefuse when the chance of being wrong times the cost of an error is higher than the cost of handing the case to a person.
A model should refuse when its calibrated confidence \(p\) falls below \(p^{*}=1-R/W\), where \(W\) is the cost of a wrong answer and \(R\) is the cost of a referral. Above that line answering has the lower expected cost, below it referring does, and the line is fixed by the two costs rather than by a round number.
The comparison
Take a wrong answer costing 50 and a referral costing 8. The threshold is \(1-8/50=1-0.16=0.84\). At confidence 0.9, answering has expected cost \((1-0.9)\times 50=0.1\times 50=5\), while referring costs 8, so answer. At confidence 0.7, answering costs \(0.3\times 50=15\), so refer. At 0.84 the two are equal: \(0.16\times 50=8\). The same arithmetic works at any scale: if the error cost is 500 and the referral cost is 80, the ratio is still \(80/500=0.16\), so the threshold is still 0.84.
The natural guess is that refusing is always safer. It is safer only in the narrow sense that it avoids a wrong answer; it is not cheaper. With the same costs, a model at confidence 0.99 faces expected cost \(0.01\times 50=0.5\) if it answers, against 8 if it refers, so refusing spends 7.5 more per case to avoid a risk already worth 0.5. Abstention buys risk reduction at a price, and when that price is above the expected loss it is the worse choice. The guess also fails at the other end: if a referral costs more than an error, the formula gives a negative threshold, which says never refuse, because every refusal costs more than the mistake it avoids.
The derivation is short. Answering costs \((1-p)W\) in expectation because a right answer costs nothing; referring costs \(R\) whatever happens. Setting \((1-p^{})W=R\) gives \(1-p^{}=R/W\) and so \(p^{*}=1-R/W\). Only the ratio \(R/W\) matters, so the costs can be in any common unit. Doubling the referral cost from 8 to 16 with the error cost still 50 gives \(1-16/50=1-0.32=0.68\), so the threshold falls from 0.84 to 0.68; the threshold moves slowly in the costs, which is why a rough estimate of each cost is enough.
| Referral cost divided by error cost, \(R/W\) | Threshold \(p^{*}\) | What it says |
|---|---|---|
| 1/5 | 0.8 | Answer above 0.8; refer below |
| 1/10 | 0.9 | Answer above 0.9; refer below |
| 1 | 0 | Never refer, since confidence cannot fall below 0 |
| 2 | -1 | Never refer; the threshold is negative |
Where the round numbers fail
The familiar 0.9 is not a principle. It is exactly right when an error costs ten times a referral, because \(1-1/10=0.9\). On the 50 and 8 numbers the threshold is 0.84, and forcing 0.9 refuses cases between 0.84 and 0.9 that are cheaper to answer. At confidence 0.86, answering costs \(0.14\times 50=7\), while referring costs 8, so the refusal loses 1 per case. A threshold of 0.95 is right only when \(R/W=0.05\), meaning a referral costs one twentieth of an error; that is a claim about the process, not a safe default. Most disputes about 0.9 against 0.95 are disputes about these two costs conducted without either being written down.
Coverage and the referral path
Raising the threshold lowers the error rate on what is answered and lowers coverage, the share answered alone. Neither number means much without the other: a system with a low error rate on a tiny share may be worse than one that answers everything at a slightly higher error rate. The threshold picks a point on that trade, and the costs decide where on the trade to sit. A single threshold is also not the same as the per-case rule: the rule gives the right decision for a case with a known confidence, while the best threshold for a whole stream depends on how many cases sit near the line and on how confidences are distributed.
The arithmetic stops being true when the referral is not real. If refused cases go to a long queue, the cost \(R\) includes the delay and is much higher than the reviewer's few minutes, so the threshold should be lower than the naive calculation. If refused cases go to someone who approves them in batches without reading, \(R\) is close to \(W\), the threshold is near zero, and the mechanism is theatre. If the person is not better at the task than the model, abstention is not the right tool at all: the honest moves are to improve the model, to answer with a wider output that admits the uncertainty, or to decline the use case.
Finally, \(p\) has to be a rate. The derivation treats confidence as the probability of being right, so an uncalibrated model gives a threshold applied to a number that does not mean what the formula assumes, and the realised error rate on answered cases comes out above the design. Calibration is what makes the cost comparison valid; without it, the threshold is a round number wearing a derivation's clothes.
Worth keeping
- Answering beats refusing exactly when the expected cost of a wrong answer, the chance of error times the error cost, is below the cost of a referral.
- The threshold is one minus the referral cost divided by the error cost, so it depends only on that ratio.
- Refusing is not free, and if a referral costs more than an error the threshold is negative, which means always answer.
Where this goes next
This question is one step of When a Model Is Sure and Wrong, a course that works through its subject from the beginning. The page that teaches this part of it is Letting It Say Nothing, and it reads the whole way through whether or not you have an account.
What an account changes is the practice: problems standing under every idea, marked with a reason rather than a tick, in a copy of the course that is yours.
Read that pageThe whole courseOther questions people ask about this
- Why can a completely made-up answer come with high confidence?Because the confidence score measures how likely the words are, and a fluent invention is built from likely words, not because the fact was checked.
- When a model says it is 90% sure, what is that 90% a probability of?It is the share of the model's score mass given to that option out of the options it was offered, not the chance that the answer is correct.
- Why doesn't temperature scaling change which answer the model picks?Temperature scaling changes how confident the model sounds, not which answer it ranks first, because dividing every score by the same positive number preserves their order.
- Why can't I fit temperature on the same data I trained the model on?Because training predictions are already too confident, fitting temperature there confirms that confidence instead of correcting it.
- Why does the model get stuck repeating the same sentence over and over?The model is not stuck in a memory loop: once it has written a phrase, that phrase is in its context and makes writing it again more likely, so a decoder that always picks the top word cannot leave.
- Why does diffusion add Gaussian noise instead of any random corruption?Diffusion adds Gaussian noise because Gaussian errors combine exactly: each step keeps the spread fixed and gives a closed-form noisy picture at any later step, which makes the reverse step learnable.