What Temperature Actually Does

Last timeSearching Wider

Dividing the scores by a constant before the softmax rescales every log-ratio between tokens by the same factor. Both limits are worth knowing, and neither end is creative.

Every generation interface has a temperature setting, and the usual explanation

is that it controls creativity. It controls one arithmetic operation, and

everything else follows from that operation.

FIG 1Where the constant goes

: the score the model produced for token i
: the temperature, a positive constant that is the same for every token

The only change from the ordinary softmax is the division inside both exponentials. It happens on the scores, before anything is turned into a probability, and there is no way to reproduce it by adjusting the probabilities afterwards.

What it does to comparisons

Take any two tokens and look at the ratio of their probabilities. The

normalising sum cancels, and what is left is an exponential of the score gap.

FIG 2The ratio between two tokens, under temperature

: the score gap, which the model decided and the temperature never touches
: divides that gap, so it raises every ratio in the vocabulary to the power of one over the temperature

This is the whole effect in one line. At a temperature of one half every ratio is squared, so a token that was three times as likely becomes nine times as likely. At a temperature of two every ratio has its square root taken. The model's ordering survives untouched at any temperature.

FIG 3The leading token, as the temperature is turned

TT = 1

probability of the leading token

Turn the temperature down and the curve steepens towards a step: any gap at all becomes near certainty. Turn it up and the curve flattens towards one half, meaning the model's opinion about which token is better stops making much difference to what comes out.

The two limits

Both ends of the dial are worth knowing exactly, because both are used.

As $T$ falls towards zero, the gaps are divided by a vanishing number, so the

largest score dominates completely and the top token takes all the probability.

Temperature zero is therefore not a separate mode with its own code path: it is

the limit of this formula, and it is greedy decoding.

As $T$ grows large, every score is divided into insignificance, the exponentials

all approach one, and the distribution approaches uniform over the whole

vocabulary. At that end the model has been switched off and what remains is a

random token generator over fifty thousand items.

The lesson stops here

3 more paragraphs to go

You have read the opening. The rest of the argument, the problems that check whether it landed, and the lines worth keeping at the end all come with a plan.

The first lesson of every course in the library reads the whole way through, free, so you can see exactly what the rest of them are.

See the plan The contents

This is the reading half

Starting the course gives you your own copy of it. Every idea on every page has problems standing under it, marked with a reason rather than a tick, and any sentence you do not believe can be opened and argued with. None of that can happen on a page nobody owns.

The contents

What Temperature Actually Does

What it does to comparisons

The two limits

3 more paragraphs to go

This is the reading half

Read alongside