What a Perplexity of Twelve Means

Last timeWhat Minimising It Is Really Doing

Perplexity is the loss with the logarithm undone, which turns it into a count of effective choices. It is easy to read and easy to quote misleadingly, and both halves matter.

The loss is a good quantity to optimise and an awkward one to talk about. Nobody

has an intuition for 2.48. Perplexity is what happens when you undo the

logarithm that made the loss convenient in the first place.

FIG 1Perplexity is the loss with the logarithm removed

: the average loss per prediction, in natural units
: the probability the model gave to the outcome that actually occurred at position n
: how many predictions were scored

Note where the average sits. The mean is taken over logarithms and then exponentiated, so perplexity is a geometric mean of the predicted probabilities, inverted. One prediction of near zero probability pulls the whole figure down in a way an ordinary average would not.

Why it reads as a number of choices

Take a model that has learned nothing beyond the size of the vocabulary and

spreads its probability evenly over $K$ options. Each prediction has probability

$1/K$ , so the loss is $\log K$ and the perplexity is exactly $K$ . That is the

anchor point for the whole interpretation: a perplexity of 12 means the model is

as uncertain as something choosing evenly between twelve options, even though

its actual distribution is nothing of the kind.

FIG 2Perplexity against loss

effective number of choices

The steepness on the right is why early training looks so dramatic when reported as perplexity and so gentle when reported as loss. Both are the same run. The right-hand end, near 10.8, is where a model over a fifty thousand word vocabulary starts before it has learned anything.

Differences in loss are ratios in perplexity

Because the two are related by an exponential, an additive change in the loss is

a multiplicative change in perplexity. A drop of $\log 2 \approx 0.693$ always

halves perplexity, whether you start at 100 or at 4. This is the practical

reason loss is quoted to three decimals: the third decimal is worth a tenth of a

percent of the effective branching factor.

The lesson stops here

3 more paragraphs to go

You have read the opening. The rest of the argument, the problems that check whether it landed, and the lines worth keeping at the end all come with a plan.

The first lesson of every course in the library reads the whole way through, free, so you can see exactly what the rest of them are.

See the plan The contents

This is the reading half

Starting the course gives you your own copy of it. Every idea on every page has problems standing under it, marked with a reason rather than a tick, and any sentence you do not believe can be opened and argued with. None of that can happen on a page nobody owns.

The contents

What a Perplexity of Twelve Means

Why it reads as a number of choices

Differences in loss are ratios in perplexity

3 more paragraphs to go

This is the reading half

Read alongside