The Model Does Not Write Anything

A language model produces one number per token in its vocabulary and stops. Everything about which word appears is decided afterwards, by a separate algorithm with its own settings.

Ask a language model to continue a sentence and it returns a list of numbers,

one for every token it knows about. Fifty thousand numbers, all positive, adding

to one. No word has been chosen. Whatever happens next is a different piece of

software making a decision, and that piece of software is the subject of this

course.

FIG 1Where the model ends and the decoder begins

Everything that was trained lives in the second box. Everything people adjust at generation time lives in the fourth. Confusing the two is the most common source of misattributed behaviour.

What is actually in that distribution

At a typical position most of the probability sits on a few tokens and the rest

is spread thinly over everything else.

FIG 2A representative step, by how many tokens hold each share

An illustration of the usual shape rather than a measurement from one model. The last slice is the interesting one: eight percent spread across fifty thousand tokens, most of which would be nonsense here. Keeping it is what makes text varied, and it is also the entire supply of ways for generation to go off the rails.

The shape changes constantly. After a phrase like the start of a quoted proper

noun, one token may hold ninety-nine percent. At the beginning of a new sentence,

no token may exceed five percent. A decoder that treats these two situations

identically is making a mistake, which is the argument the truncation lesson

turns into a method.

Sequences are products

The probability the model assigns to a whole continuation is the product of the

per-step probabilities, each conditioned on everything before it.

FIG 3The probability of a continuation

: the token at position i
: the number the model produced for that token at that position, given everything already written

Every factor is below one, so the product falls fast. A forty token sentence that the model is unusually happy with might have a probability around ten to the minus twenty, which sounds like a rejection and is in fact a strong endorsement.

FIG 4How the probability of a continuation falls with its length

each step a comfortable 0.9each step a merely plausible 0.6

Both curves are of a model doing well. After sixty tokens the upper one is around 0.0018 and the lower one is near ten to the minus thirteen. Any absolute reading of these numbers is meaningless; only the comparison between two continuations of the same length says anything.

The same numbers, four different outputs

Given one distribution, the decoder has several reasonable things it could do.

FIG 5Four decoders at the same position, given the same distribution

step	probability of the token chosen	its rank in the list	probability mass it ignored	what happened
1	0.42	1	0.58	Take the most likely token. Repeatable, and it throws away every alternative including the ones that were nearly as good.
2	0.11	4	0	Sample from the whole distribution. This time the fourth most likely token came out, and nothing was excluded, including the fifty thousand tokens in the tail.
3	0.19	2	0.19	Sample after keeping only the top five. The tail is gone and the remaining five were reweighted to sum to one.
4	0.42	1	0.27	Sample after keeping the smallest set whose mass reaches 0.73. Here that set was three tokens, and the top one happened to come out anyway.

4 steps

One model, one position, four outputs. Anyone reporting what a model said is also reporting, whether they know it or not, which of these rows they ran.

No going back

The token that comes out is appended to the context and the model is run again.

It now predicts conditional on a choice that the decoder made, which the model

itself would perhaps not have made. Nothing revisits it.

This is why a single unlucky token matters so much more than its probability

suggests. Sample a slightly odd word at the start of a sentence and every

subsequent step is conditioned on it: the model will do its competent best to

continue the sentence it now finds itself in, and forty tokens later the

paragraph is about something else. Search methods, which the next two lessons

cover, are attempts to buy back some of the ability to reconsider.

What to hold on to

The model produces one distribution per position and chooses nothing. The

probability of a continuation is the product of those per-step numbers, so it is

tiny for any continuation and useful only as a comparison. Each emitted token

becomes permanent context. Everything else in this course is about the algorithm

sitting between those distributions and the text.

Recap

The model emits a full distribution over the vocabulary at every position, and nothing in it selects a word. Selection is a separate algorithm, which means a change in the output may be a change in the decoder rather than in the model.
The probability of a sequence is the product of the per-step probabilities, so every continuation of any length is individually very unlikely, and comparing two of them is the only meaningful use of the number.
Each token that is emitted is fed back in as context, so a choice cannot be reconsidered later. The decoder is committing at every step, with no way to undo.

This is the reading half

Starting the course gives you your own copy of it. Every idea on every page has problems standing under it, marked with a reason rather than a tick, and any sentence you do not believe can be opened and argued with. None of that can happen on a page nobody owns.

The contents

NextThe Likeliest Word Is Not the Likeliest Sentence →