How a Model Picks Its Next Word
Guessing Ahead Without Changing the Answer
Last timeForcing It Into a Format
A small model can guess several tokens ahead and the large one checks them all in a single pass. The accept rule is built so that the text that comes out is distributed exactly as before.
Everything so far has been about which token to choose. This lesson is about how
long it takes to get it, and it turns out the two can be separated completely:
there is a way to go several times faster that provably does not change what
comes out.
The machine is mostly idle
Producing one token means reading every weight of the model out of memory and
doing a comparatively small amount of arithmetic with each one. The time goes on
the reading. That has an odd consequence: scoring five positions in one pass
costs barely more than scoring one, because the weights are read the same number
of times.
So one token at a time is the worst possible use of the hardware, and the only
reason it is done that way is that each token depends on the one before it.
Let something cheap guess
Use a much smaller model to write the next few tokens. It is wrong often enough
to matter and fast enough not to. Then hand its guesses to the large model,
which scores every position in a single pass and thereby learns what it would
itself have said at each one.
The lesson stops here
5 more paragraphs to go
You have read the opening. The rest of the argument, the problems that check whether it landed, and the lines worth keeping at the end all come with a plan.
The first lesson of every course in the library reads the whole way through, free, so you can see exactly what the rest of them are.
See the planThe contentsThis is the reading half
Starting the course gives you your own copy of it. Every idea on every page has problems standing under it, marked with a reason rather than a tick, and any sentence you do not believe can be opened and argued with. None of that can happen on a page nobody owns.
The contents