The Model You Must Hold and the Model You Run
Last timeTraining Something That Makes Choices
Serving a sparse model means holding every copy on hardware that will use two of them per token, moving tokens between machines twice per layer, and needing a large batch before any of it pays.
The arithmetic saving is real. Everything else about serving one of these models
is harder than serving a dense one, and the difficulties are specific enough to
be worth going through one at a time.
Everything has to be there
A token visits two copies out of sixty four. It is tempting to conclude that
sixty two copies could be somewhere cheaper, loaded when needed. They cannot,
for a reason that is structural rather than an implementation shortcoming.
Which copies a token will visit is a result of running the chooser, which
happens in the middle of processing that token. Finding out costs a
multiplication, and acting on the answer has to happen immediately, because the
next layer is waiting. There is no window in which a copy could be fetched from
anywhere slower than memory. And the next token will choose differently, so
nothing can be usefully cached either.
| held | touched per token | machines needed to hold | |
|---|---|---|---|
| a dense model | 8 | 8 | 1 |
| a medium sparse model | 47 | 13 | 6 |
| a large sparse model | 671 | 37 | 84 |
Spreading the copies
Eighty machines holding a complete copy of the model each would be absurd, so
the copies are divided up: each machine holds a subset, and the memory per
machine falls in proportion. This is the only arrangement in which a very large
sparse model fits anywhere.
The consequence is that a token now has to physically reach the machine holding
its chosen copy.
The lesson stops here
4 more paragraphs to go
You have read the opening. The rest of the argument, the problems that check whether it landed, and the lines worth keeping at the end all come with a plan.
The first lesson of every course in the library reads the whole way through, free, so you can see exactly what the rest of them are.
See the planThe contentsThis is the reading half
Starting the course gives you your own copy of it. Every idea on every page has problems standing under it, marked with a reason rather than a tick, and any sentence you do not believe can be opened and argued with. None of that can happen on a page nobody owns.
The contents