Answers · How models work
When a model says it is 90% sure, what is that 90% a probability of?
In shortIt is the share of the model's score mass given to that option out of the options it was offered, not the chance that the answer is correct.
Of the options it was handed, and conditional on one of them being the answer. It is not the chance the answer is correct. The number is what you get when a list of raw scores is forced to sum to one, and forcing a list to sum to one tells you nothing about whether the list was any good.
- the raw score for option i, an unconstrained real number that can be negative
- that score exponentiated, which makes it positive and turns a one-unit lead into a fixed multiplicative advantage
- the same quantity totalled over every option offered, which is what forces the outputs to sum to one
Take three options with scores 4.1, 1.2 and 0.7. Exponentiate each: , , . Those total 65.67. Divide each by the total: , , . The model reports 0.92. The whole of that 0.92 is the top score being about eighteen times the second, since , with the exponentiation turning that ratio into a share of the total.
| scores | exponentiated | total | reported | actually right? |
|---|---|---|---|---|
| 4.1, 1.2, 0.7 | 60.34, 3.32, 2.01 | 65.67 | 0.92, 0.05, 0.03 | yes |
| 1.4, 1.3, 1.2 | 4.06, 3.67, 3.32 | 11.04 | 0.37, 0.33, 0.30 | no |
| 3.9, 1.1, 0.9 | 49.40, 3.00, 2.46 | 54.87 | 0.90, 0.05, 0.04 | no |
The natural reading is that 0.92 means ninety-two times in a hundred this answer is right. The formula cannot deliver that, and the obstacle is the denominator. It divides by the sum over the options offered, so what comes out is a probability conditional on one of those options being the answer. There is no entry for none of them. Ask a model to choose among three diagnoses when the patient has a fourth, and the three numbers still sum to one and one wrong option still takes most of the mass. The model has not failed; it answered which of these, not whether any of these. The first and third rows of the table carry the rest of the point. A reported 0.92 and a reported 0.90 are the same number to a reader, one is right and one is wrong, and the scores behind them are the model's only judgement. The formula has no access to whether the pattern that produced those scores was true. A confidence of 0.9 describes the shape of a score list, not the world.
The same lead, a longer list
Change how many options are offered and the identical internal judgement becomes a different number. Let the leading score be 3 clear of the others and offer two options, the second at zero. Then , the total is 21.09, and the reported figure is . Keep the lead at 3 but offer ten options, the other nine all at zero. The total becomes and the reported figure is . Same lead, same evidence, 0.95 or 0.69 depending only on the length of the list. A question with four distractors manufactures a different confidence from an otherwise identical question with one, and nothing about what the model knows has changed.
One token is not one claim
A language model does not score answers. It scores the next token, over a vocabulary of around a hundred thousand, and a twenty-token reply carries twenty such numbers. Multiplying them gives the probability of that exact sequence of tokens. If each token sits at 0.9, the product is . That 0.12 is a real quantity and it is the probability of that wording, not the probability of the claim. Rephrase the same answer and you have a different sequence with a different number, usually smaller for a longer reply, because every extra token multiplies in another factor below one. Averaging per token instead of multiplying is the usual repair, and the average is still not a chance of being right.
The ordering survives, the gaps do not
Training rewarded putting the right option above the wrong ones, so the ranking is usually good, and anything that only needs a ranking works: taking the top answer, returning a shortlist, rejecting the bottom of the list. Nothing in training constrained the size of the gaps. The loss falls as the correct score is pushed higher, without limit, so a model that is right on ninety questions in a hundred can report 0.99 on all of them and pay nothing for the exaggeration. Route every case above 0.9 straight through and staff on the belief that one in a hundred of those will be wrong. If ninety of the hundred are right, ten are wrong, which is ten times the staffing the number implied.
It becomes a rate only after somebody measures it. Score a few thousand held-out questions, bucket them by reported confidence, and count the fraction actually right in each bucket. If the 0.9 bucket is right about ninety per cent of the time, the number is calibrated for that distribution and may be treated as a rate; if it is not, a monotone remapping fitted on the same held-out data will usually make it so. Two limits survive the repair. Calibration is a property of the distribution measured, so a shift in topic, wording or option count can undo it. And a rate is a frequency over answers like this one, not a statement about this answer.
Worth keeping
- A reported 0.9 is a probability over the options the model was given, conditional on one of them being the answer, so an answer that was never on the list cannot be represented by any of the numbers.
- The same score lead reports 0.95 with two options and 0.69 with ten, so the figure tracks the length of the list as well as the evidence.
- The ranking of options is usually reliable while the size of the gaps is not, so the number is safe for picking a top answer and unsafe as a failure rate until it has been measured.
Where this goes next
This question is one step of When a Model Is Sure and Wrong, a course that works through its subject from the beginning. The page that teaches this part of it is What That Number Actually Is, and it reads the whole way through whether or not you have an account.
What an account changes is the practice: problems standing under every idea, marked with a reason rather than a tick, in a copy of the course that is yours.
Read that pageThe whole courseOther questions people ask about this
- Why can a completely made-up answer come with high confidence?Because the confidence score measures how likely the words are, and a fluent invention is built from likely words, not because the fact was checked.
- Why doesn't temperature scaling change which answer the model picks?Temperature scaling changes how confident the model sounds, not which answer it ranks first, because dividing every score by the same positive number preserves their order.
- When should a model refuse to answer instead of guessing?Refuse when the chance of being wrong times the cost of an error is higher than the cost of handing the case to a person.
- Why can't I fit temperature on the same data I trained the model on?Because training predictions are already too confident, fitting temperature there confirms that confidence instead of correcting it.
- Why does the model get stuck repeating the same sentence over and over?The model is not stuck in a memory loop: once it has written a phrase, that phrase is in its context and makes writing it again more likely, so a decoder that always picks the top word cannot leave.
- Why does diffusion add Gaussian noise instead of any random corruption?Diffusion adds Gaussian noise because Gaussian errors combine exactly: each step keeps the spread fixed and gives a closed-form noisy picture at any later step, which makes the reverse step learnable.