The Ceiling You Cannot Code Around
Last timeHow Many Bits a Number Needs
One request writing one word at a time must read the whole model from memory for every word, and that alone fixes the fastest it can possibly go.
Here is the central fact of the course, and it is worth stating before any of the
reasoning. To produce one word, a model reads all of its weights. Not some of
them, not the relevant ones. Every table in every layer is multiplied by the
numbers passing through, so every weight has to make the journey from memory to
the multipliers, and it has to make it again for the next word.
That single sentence, combined with the bandwidth figure from the first lesson,
settles how fast a conversation can possibly go.
- the number of weights in the model
- the bytes each weight occupies in whatever format it is stored in
- the bytes the memory can deliver each second
- words a second, at most, for a single request producing them one at a time
Checking that the arithmetic really is irrelevant
It is easy to accept this too quickly, so it is worth doing the other half of the
sum. Every weight contributes one multiplication and one addition, so a model of
seventy billion weights performs about a hundred and forty billion operations to
produce a word. A chip advertising three hundred trillion operations a second
gets through that in under half a millisecond.
| step | weights, in billions | gigabytes read per word | milliseconds of movement | milliseconds of arithmetic | what happened |
|---|---|---|---|---|---|
| 1 | 7 | 14 | 4.7 | 0.05 | A small model, comfortably faster than anyone reads. The arithmetic is already a rounding error. |
| 2 | 13 | 26 | 8.7 | 0.09 | Doubling the weights doubles both columns, so the ratio between them never moves. |
| 3 | 70 | 140 | 46.7 | 0.47 | The common serving size. Twenty-one words a second, with the multipliers busy one percent of the time. |
| 4 | 400 | 800 | 266.7 | 2.7 | A frontier size on one machine. Under four words a second, which is why models this large are always spread over many chips. |
Forty-seven milliseconds against half a millisecond. The chip's headline figure,
the one it is sold on, governs one percent of the time taken. If a vendor doubled
its arithmetic throughput tomorrow and left the memory alone, a single
conversation would run at exactly the same speed.
The lesson stops here
4 more paragraphs to go
You have read the opening. The rest of the argument, the problems that check whether it landed, and the lines worth keeping at the end all come with a plan.
The first lesson of every course in the library reads the whole way through, free, so you can see exactly what the rest of them are.
See the planThe contentsThis is the reading half
Starting the course gives you your own copy of it. Every idea on every page has problems standing under it, marked with a reason rather than a tick, and any sentence you do not believe can be opened and argued with. None of that can happen on a page nobody owns.
The contents