The Two Kinds of Work
A chip that runs a model multiplies numbers and moves numbers, and the second is hundreds of times slower, which quietly decides almost everything else.
A chip that runs a trained model does two separable things, and almost every badly
wrong estimate of what serving costs comes from collapsing them into one. The
first is arithmetic: multiplying two numbers and adding the result to a running
total, which is very nearly the only operation a model performs. The second is
movement: fetching those numbers out of the memory they are stored in and
delivering them to the small fast region where the multiplying happens.
Both have a top speed. Both speeds are published. And the two are nothing like
each other.
The two numbers on the box
Take a representative accelerator of the current generation. It will advertise
something near three hundred trillion multiply-and-add operations per second in a
half-size number format, and something near three terabytes per second of memory
bandwidth. The first is how fast it can calculate, the second how fast it can be
fed.
Divide one by the other and you get a number that describes the machine rather
than any job: three hundred trillion over three trillion is one hundred. The chip
can perform about a hundred operations in the time it takes to fetch a byte. Any
job carrying more arithmetic than that per byte will be held up by the arithmetic.
Any job carrying less will be held up by the fetching, and the expensive
multiplying units will sit idle.
- the operations the chip can perform each second, as advertised
- the bytes it can read out of its own memory each second, also as advertised
- operations per byte, meaning how much arithmetic a job has to carry for every byte it reads if the multiplying units are to be kept busy
That number is not a quirk of one product. It has drifted upward for thirty years,
because the multipliers that can be packed onto a chip have grown much faster than
the rate at which numbers can be delivered to them. A machine from a decade ago
sat nearer ten. The direction of travel is the part worth remembering: as hardware
improves, movement becomes relatively more expensive, not less.
Operations per byte of a real piece of work
Now the job. A layer of a model is a large table of weights, and pushing one list
of numbers through it means multiplying the list by the table: for every weight,
one multiplication and one addition. A table of a million weights therefore costs
two million operations and requires every weight to be read once. At two bytes a
weight, that is two million bytes moved to do two million operations. Exactly one
operation per byte.
One against a hundred. Writing a single word of an answer, the model reaches about
one percent of what the chip could calculate. The same sum points at the way out:
push not one list through the table but many at once, and the table is read once
and used for all of them.
- the width of the table of weights, so that the table holds n squared of them
- how many separate lists of numbers are pushed through the table together
- the operations: one multiply and one add for every weight, repeated for each of the lists
- the bytes read, which is the whole table once at two bytes to a weight, however many lists are waiting
- operations per byte, which comes out as the number of lists and nothing else
That cancellation is easy to miss. A small model and a huge one, each serving one
request at a time, waste the same ninety-nine percent. The huge one is slower in
absolute terms, having more bytes to read, but the inefficiency is about how many
requests are in flight at one moment, and a later lesson is given over to it.
The shape of the limit
The two limits apply at the same time rather than in sequence, so a piece of work
takes the larger of the two times it implies, not their sum. Plot the rate you can
achieve against the arithmetic the work carries per byte and the picture has a
characteristic shape: a straight rise, where you are getting everything memory can
deliver, then a flat ceiling, where you are getting everything the multipliers can
do. The corner sits at the machine's balance point.
This picture is old and not specific to models. It was named the roofline and
proposed for any program on any multicore machine, long before a language model
existed. Its value is that it refuses to let you call a program fast or slow
without saying which wall it is standing against, because the remedy differs
completely. Against the ceiling you need a better algorithm or more hardware.
Against the slope, extra arithmetic is free.
| operations carried per b | percent of the arithmeti | |
|---|---|---|
| writing one word for one | 1 | 1 |
| writing one word for six | 16 | 16 |
| writing one word for a h | 100 | 100 |
| reading a two thousand w | 2000 | 100 |
A first pass over the rest of the course
Those four rows are most of what a serving system does, and they differ by three
orders of magnitude. Reading a prompt is the comfortable case: the whole prompt
goes through each table at once, so the table is read once and used thousands of
times. Writing the answer is the opposite case, and it is also the case that
takes the time, since a reply of five hundred words is five hundred separate
passes through the entire model.
Everything in production serving that looks like a trick is an attack on that
first row. Shrinking the number format halves the bytes without touching the
operations. Grouping requests multiplies what the work carries per byte by the
size of the group. Letting a small model guess several words so the large one can
check them in a single pass converts a run of single words into one wider piece of
work. Different mechanisms, one target.
Where the bytes actually travel
That is the frame for everything that follows. A job has two costs, a machine has
two speeds, dividing the speeds gives one threshold, and making a model cheap to
run is largely the business of taking work that sits at one operation per byte and
getting it up somewhere near a hundred.
Recap
- Every piece of work has an arithmetic cost and a movement cost, and the time it takes is set by whichever of the two is larger, never by the two of them added together.
- A machine's two published speeds divide to give one number, the operations it can perform per byte it can fetch, and that number is a line which any given job either clears or does not.
- A model answering one request a word at a time carries about one operation per byte it reads, roughly a hundred times short of the line, so it is a movement problem wearing the costume of a computing problem.
This is the reading half
Starting the course gives you your own copy of it. Every idea on every page has problems standing under it, marked with a reason rather than a tick, and any sentence you do not believe can be opened and argued with. None of that can happen on a page nobody owns.
The contents