Answers · How models work
Why can a completely made-up answer come with high confidence?
In shortBecause the confidence score measures how likely the words are, and a fluent invention is built from likely words, not because the fact was checked.
A made-up answer can come with high confidence because the number attached to it scores how likely the words were, not whether the claim is true. An invention assembled out of likely words gets a high product, and nothing in that calculation asks whether the thing named exists.
Take a request for the lead author of a paper that does not exist. The model is not handed a field marked 'exists: no'. It is handed a prefix, 'the lead author of that paper is', and the likely continuations are surnames. Suppose the four chosen tokens scored 0.98, 0.97, 0.99 and 0.99. Their product is 0.98 × 0.97 × 0.99 × 0.99 = 0.9317, so the reported confidence is about 0.93. The number is correct about the only thing it was computed from: those words were likely under the model's distribution. A true but awkward name can score lower. If one correct token had probability 0.40 where the fluent invention had 0.98, the same product would be 0.40 × 0.97 × 0.99 × 0.99 = 0.3803, and the confidence would fall to about 0.38. Fluency, not truth, is what moves the number.
The natural guess is that a high number means the model checked the fact and found it solid. It did not. Existence was never part of the next-word question, so the model has no state of uncertainty about it to report. A fluent fabrication can be more probable than a clumsy truth, which is why the score can be highest on exactly the answers that most need checking. The probability is a statement about strings under a training distribution, not a statement about the world.
Why calibration leaves it standing
Calibration adjusts the mapping from scores to outcomes so that answers given 90 per cent are right about 90 per cent of the time. That is a statement about averages over a bucket, and a fabrication is one answer inside a bucket. Temperature scaling fits a single number to held-out data. If fabrications are 3 per cent of that data, the fit is set by the other 97 per cent, and the invented case is rescaled by whatever number the majority asked for. Sampling several times helps only when the invention is unstable. Where one invented answer is the single most likely continuation, five samples return the same invention five times.
Read the first term as the error the number reports and the second as the error it cannot see. With p=0.93, the reported error is 1-0.93=0.07. Add a measured invention floor g=0.03 and the error to plan for is 0.07+0.03=0.10, not 0.07. With referral cost one fifth of an error cost, the old threshold compares 1-p with 0.20 and gives p=0.80. Including the floor compares (1-p)+0.03 with 0.20, so 1-p must exceed 0.17 and the threshold is p=0.83. The shift is small. The useful part is that even a threshold of 0.99 leaves the 0.03 floor untouched.
| Question | Reported confidence | Same answer in five samples | Actually true |
|---|---|---|---|
| Well documented date | 0.96 | 5 of 5 | yes |
| Date the model half knows | 0.71 | 3 of 5 | no |
| Author of a real paper | 0.88 | 4 of 5 | no |
| Author of a paper that does not exist | 0.93 | 5 of 5 | no |
A reliability curve has the same blind spot for the same reason. It buckets predictions and measures outcomes, so a model can sit perfectly on the diagonal while every bucket contains a few confidently invented answers. Calibration is a claim about the average of a bucket; a fabrication is a claim about one answer. That is why a well calibrated system can still be confidently wrong at a rate that matters, and why the rate has to be measured by checking a sample of confident answers against sources rather than read off the curve.
What changes the answer
Repairs that work change what the model is answering from. Put the source text in the request and ask for an answer from that text alone: the question becomes what the passage says, which is a question the model can be wrong about in a detectable way. Require a citation and test it mechanically against the supplied text. The quoted span either appears there or it does not, and that test needs no model. For a thing with an identity, a paper, a person, a part number, look it up, because existence is usually a database query rather than a judgement. A second pass whose only job is to mark each claim as supported or unsupported catches much of what the first pass invented.
None of this makes the confidence number useless. It remains the right thing for ranking ordinary uncertainty and for setting a review threshold, and an uncalibrated system cannot do either. It stops being enough when the existence of the answer is the question. There the number is silent, the sample may be unanimous, and the only reliable evidence is a check against an external source. Treat an unsupported claim as an error whatever its number says.
Worth keeping
- A high confidence number is a statement about how likely the words were, not about whether the claim is true.
- Calibration fixes averages and cannot remove a floor of confident invention.
- Supplying source text and checking citations catches inventions that no confidence threshold can see.
Where this goes next
This question is one step of When a Model Is Sure and Wrong, a course that works through its subject from the beginning. The page that teaches this part of it is The Confident Invention, and it reads the whole way through whether or not you have an account.
What an account changes is the practice: problems standing under every idea, marked with a reason rather than a tick, in a copy of the course that is yours.
Read that pageThe whole courseOther questions people ask about this
- When a model says it is 90% sure, what is that 90% a probability of?It is the share of the model's score mass given to that option out of the options it was offered, not the chance that the answer is correct.
- Why doesn't temperature scaling change which answer the model picks?Temperature scaling changes how confident the model sounds, not which answer it ranks first, because dividing every score by the same positive number preserves their order.
- When should a model refuse to answer instead of guessing?Refuse when the chance of being wrong times the cost of an error is higher than the cost of handing the case to a person.
- Why can't I fit temperature on the same data I trained the model on?Because training predictions are already too confident, fitting temperature there confirms that confidence instead of correcting it.
- Why does the model get stuck repeating the same sentence over and over?The model is not stuck in a memory loop: once it has written a phrase, that phrase is in its context and makes writing it again more likely, so a decoder that always picks the top word cannot leave.
- Why does diffusion add Gaussian noise instead of any random corruption?Diffusion adds Gaussian noise because Gaussian errors combine exactly: each step keeps the spread fixed and gives a closed-form noisy picture at any later step, which makes the reverse step learnable.