Answers · The mathematics underneath

What does a perplexity of 10 actually mean?

In shortA perplexity of 10 means the model's average loss is about 2.30 nats per prediction, equivalent to a uniform choice among ten equally likely options, not a 10 percent accuracy.

A perplexity of 10 means the model's average loss is $\ln 10$ , about 2.303 nats per prediction, which is the same uncertainty as spreading probability evenly over ten equally likely outcomes. It is not a 10 per cent accuracy rate, and it is not a literal count of ten possible next words.

\mathrm{PPL}=e^{\overline J},\qquad \overline J=\frac1N\sum_{n=1}^N -\log p_n

The arithmetic behind the ten

Take the flat case first. If a model spreads probability evenly over $K$ options, the outcome that occurs gets probability $1/K$ at every prediction. The loss there is $-\log(1/K)=\log K$ , and undoing the logarithm gives $e^{\log K}=K$ . Put $K=10$ : the loss is $\log 10=2.302585\ldots$ and the perplexity is $e^{2.302585\ldots}=10$ . That is exact, not a rounded analogy. For a real model the distribution is not flat, so 10 is the effective number of flat choices with the same geometric mean probability. Written out, $\mathrm{PPL}=1/(p_1p_2\cdots p_N)^{1/N}$ , a geometric mean, which is why a single prediction with near zero probability pulls it further than an ordinary average would. The model is not actually rolling a ten-sided die; it is being summarised by the flat die that would have scored the same.

FIG 1

average loss	perplexity	change in loss from ten
1.609438	5	-0.693147
2.302585	10	0
2.995732	20	0.693147

Because the relation is exponential, a fixed change in loss is a fixed ratio in perplexity. Lowering the average loss by $\log 2=0.693147$ multiplies perplexity by $e^{-0.693147}=0.5$ , so 10 becomes 5. Raising it by the same amount doubles it, so 10 becomes 20. The third decimal of loss is therefore not decoration: a change of 0.001 in loss changes perplexity by a factor of $e^{0.001}\approx1.001$ , about a tenth of a per cent. The same fall from 100 to 50 is also a drop of 0.693, not a drop of 50 in loss.

The wrong reading

The natural guess is that 10 means the model is choosing between ten candidates, or that it will be right 10 per cent of the time. Neither follows. Perplexity scores the probability assigned to the token that actually occurred, not whether the top guess was correct. A model that puts probability 0.9 on the right token at a prediction has perplexity $1/0.9=1.111$ at that prediction, even though the remaining 0.1 may be spread over tens of thousands of words. A model that puts 0.1 on the right token has perplexity $1/0.1=10$ , but that is one prediction, not an average, and it does not mean ten candidates were live in any literal sense. Accuracy is a thresholded count of top-1 hits; perplexity is a geometric mean of the probabilities the model gave to what happened. They can move in different directions. A model can be usually right and still have a poor perplexity if it is badly wrong, in probability terms, on the tokens that matter to the average.

A model with no learning beyond vocabulary size anchors the scale. If the vocabulary has $K$ entries and the model spreads probability evenly, its perplexity is $K$ . For a 50,257 token vocabulary that is 50,257, since $\log 50257\approx10.825$ and $e^{10.825}\approx50257$ . A perplexity of 10 is therefore far below the unlearned baseline: it means an effective branching factor of ten, not a vocabulary of ten.

Where it stops being comparable

Perplexity is defined per prediction, and the prediction unit is chosen by the tokeniser. Split the same text into smaller pieces and each piece is easier to predict, so per-piece perplexity falls while the model's grasp of the text is unchanged. A perplexity of 10 with one tokeniser is not the same measurement as a perplexity of 10 with another, even on the same text and the same model weights. To compare across tokenisers, restate the loss per character:

\mathrm{PPL}_{\mathrm{char}}=e^{\overline J\cdot(\text{tokens}/\text{characters})}

Multiplying the per-token loss by tokens per character puts the exponent on a per-character footing, and a per-character figure can be compared across tokenisers. The conversion is not cosmetic. If one tokeniser packs four characters into a token and another packs two, the same per-character loss gives per-token losses in the ratio 4:2, so the per-token perplexities differ by an exponent of 2, not by a few per cent. The corpus and the context length must match as well. Perplexity on news and perplexity on code are different measurements, and the same model scored with two thousand tokens of context will usually beat itself scored with two hundred. A perplexity quoted without its tokeniser, its corpus and its context length is not a measurement of anything.

Worth keeping

A perplexity of 10 corresponds to an average loss of about 2.303 nats per prediction, because the natural logarithm of 10 is 2.302585.
A fixed drop of 0.693 in average loss halves perplexity, so 10 becomes 5.
Perplexities are comparable only when the tokeniser, corpus and context length are the same.

Where this goes next

This question is one step of The Mathematics of Cross Entropy, a course that works through its subject from the beginning. The page that teaches this part of it is Reading the Number, and it reads the whole way through whether or not you have an account.

What an account changes is the practice: problems standing under every idea, marked with a reason rather than a tick, in a copy of the course that is yours.

Read that page The whole course