Basis Lab

Answers · How models work

What does temperature actually do to the next-word probabilities?

In shortIt divides the model's raw scores by a constant before they are turned into probabilities, so every ratio between two candidate words is pushed further apart or pulled closer together.

Temperature divides the model's raw scores by a constant before the softmax is taken, so every ratio between two token probabilities comes out raised to the power of one over the temperature. It never edits the probabilities themselves: it changes the arithmetic that produces them.

pi=ezi/T∑jezj/Tp_i = \frac{e^{z_i / T}}{\sum_j e^{z_j / T}}

The only difference from an ordinary softmax is the division inside the exponentials. It happens on the scores, which are unnormalised numbers the final layer produced for each token, comparable to one another but not yet probabilities. Nothing downstream can reproduce it, because by the time you have probabilities the information about how far apart the scores were has already been spent.

Take two tokens whose scores differ by 1.0, say 2.0 and 1.0. At temperature 1 the exponentials are e2.0=7.389e^{2.0} = 7.389 and e1.0=2.718e^{1.0} = 2.718, which sum to 10.107, so the pair come out at 0.731 and 0.269 and the ratio between them is 2.718. Halve the temperature and the scores become 4.0 and 2.0: the exponentials are 54.598 and 7.389, the probabilities 0.881 and 0.119, the ratio 7.389, which is 2.71822.718^2. Double the temperature and the scores become 1.0 and 0.5: the exponentials are 2.718 and 1.649, summing to 4.367, giving 0.623 and 0.377, a ratio of 1.649, the square root of 2.718. The gap of 1.0 never moved. Only the number it was divided by did.

papb=e(za−zb)/T\frac{p_a}{p_b} = e^{(z_a - z_b)/T}

The normalising sum cancels in that ratio, which is why the effect is clean: the score gap gets divided by TT and then exponentiated. Temperature 1 leaves the model exactly as it was trained. A ratio of 3 becomes 32=93^2 = 9 at temperature one half, and 3=1.732\sqrt{3} = 1.732 at temperature 2. Every comparison in the vocabulary is stretched or compressed by the same factor, and because dividing by a positive number preserves order, the ranking of tokens is identical at every positive temperature.

FIG 1The leader's probability against the score gap
0.400.580.750.931.100.01.53.04.56.0score gap between the top two tokens
T = 0.5T = 1T = 2
At a gap of 1.0 the three curves read 0.881, 0.731 and 0.623. They all pass through one half at a gap of zero, where the temperature has no ratio to work on.

The tempting picture is that temperature scales the probabilities or sprinkles noise over them. Scaling fails immediately: multiply every probability by the same constant and no ratio changes at all, and renormalising removes the constant entirely. It is a no-op. Noise fails differently, because noise on the probabilities would have to know how large each probability already was, and it would operate on the wrong object; the division sits upstream of the probabilities, on scores the model had already committed to.

What temperature changes is not which token is preferred but by how much. A gap of 1.0 is worth 2.0 at temperature one half and 0.5 at temperature 2. Raising it moves mass from the head into a tail the model already scored low, and a low score is usually there for a reason. That is the entire trade: variety bought from the tokens the model judged worst among those it had not ruled out.

The two ends

Both extremes are worth knowing exactly. As TT falls towards zero the divided gap blows up, the exponent for the leader goes to infinity relative to everything else, and it takes all the mass: temperature zero is greedy decoding, not a separate algorithm with its own code path. As TT rises, at temperature 100 the gap of 1.0 becomes 0.01, the ratio is e0.01=1.010e^{0.01} = 1.010, and two tokens sit at 0.502 and 0.498. Over a fifty-thousand-token vocabulary each would hold about 1/50000=0.000021/50000 = 0.00002 and the model's opinion would have stopped mattering altogether.

Strictly, the formula is undefined at T=0T = 0 because it divides by it, so implementations special-case that setting rather than compute it. Every numerical claim above also assumes the scores reaching the division are the model's own; a repetition penalty, a bias term or a grammar mask rewrites the gaps first, and temperature then works on the rewritten ones.

One practical detail changes the answer. Temperature is nearly always applied before truncation, and top-p decides what to keep by reading the probabilities that temperature has just flattened. Turning the heat up therefore does two things at once: it spreads mass into the tail, and it widens the surviving set, because more tokens now clear the cumulative threshold. People routinely credit the second effect to the first. Apply temperature after the cut instead, which some interfaces allow, and the second effect vanishes: the same two numbers then give noticeably different behaviour, since the candidate set was fixed before the flattening happened.

Worth keeping

  • Temperature divides the scores before the softmax, so the ratio between any two token probabilities becomes that ratio raised to the power of one over the temperature.
  • Zero temperature is the limit of the same formula and equals greedy decoding; a very large temperature drives the distribution towards uniform over the whole vocabulary.
  • No new options are created at any setting, because the scores and their ordering are fixed before the division happens.

Where this goes next

This question is one step of How a Model Picks Its Next Word, a course that works through its subject from the beginning. The page that teaches this part of it is One Knob on the Scores, and it reads the whole way through whether or not you have an account.

What an account changes is the practice: problems standing under every idea, marked with a reason rather than a tick, in a copy of the course that is yours.

Read that pageThe whole course

Other questions people ask about this

  1. Why does the model get stuck repeating the same sentence over and over?The model is not stuck in a memory loop: once it has written a phrase, that phrase is in its context and makes writing it again more likely, so a decoder that always picks the top word cannot leave.
  2. If you ban certain tokens, does that change what the model would have said?Banning tokens leaves the model's raw scores alone but changes the probabilities you sample from, so the text can change.
  3. Why isn't picking the most likely word at each step the best way to write a sentence?Because the most likely word now can lead to a much worse sentence than a slightly less likely word that sets up a better continuation.
  4. Why does diffusion add Gaussian noise instead of any random corruption?Diffusion adds Gaussian noise because Gaussian errors combine exactly: each step keeps the spread fixed and gives a closed-form noisy picture at any later step, which makes the reverse step learnable.
  5. Why does the same request cost more tokens in some languages than in others?A token is learned from a corpus, so a language that was rare in the training text is split into many more pieces for the same meaning, and the price multiplier follows.
  6. How do you turn the predicted noise into the mean of the previous image?Subtract a small scheduled fraction of the predicted noise from the current image, then rescale to undo the forward step's shrinkage.

Read alongside