You Do Not Choose One, You Multiply Four
Last timeNobody Holds the Whole Thing
The four ways of dividing training are not alternatives. They compose, and each one is assigned to the link whose speed its traffic can live with.
Five lessons have each presented one way to divide the work, and each one ended
with a limit. That is not four candidates to pick between. It is four axes, and
a real training run uses all of them at once.
| messages per step | megabytes per message | collective, not point to | link speed it needs, GB | practical degree | |
|---|---|---|---|---|---|
| same model, different da | 1 | 20 | 0 | 25 | 1024 |
| same layer, split across | 160 | 100 | 1 | 400 | 8 |
| different layers, differ | 6 | 30 | 0 | 25 | 16 |
| state split, weights gat | 3 | 60 | 1 | 400 | 64 |
The four axes
Read the rows as traffic signatures rather than as techniques. The data split
sends one large message at the end of a step, which is why a larger per-device
batch helps it. The inside-layer split sends a hundred and sixty medium
messages that nothing can hide, because the next operation in the layer is
waiting on each one. The stage split sends a handful of messages to one
neighbour each, which can be overlapped. The state split adds gathers that
scale with the model rather than the batch.
They multiply
Each axis has a practical ceiling, and all four ceilings are low. Beyond eight
or sixteen, splitting inside a layer runs out of fast cabling. Beyond sixteen
stages, the idle fraction demands a batch nobody can afford. The data split
runs out when the per-device batch falls too low to keep a device busy.
The resolution is that the degrees multiply.
The lesson stops here
4 more paragraphs to go
You have read the opening. The rest of the argument, the problems that check whether it landed, and the lines worth keeping at the end all come with a plan.
The first lesson of every course in the library reads the whole way through, free, so you can see exactly what the rest of them are.
See the planThe contentsThis is the reading half
Starting the course gives you your own copy of it. Every idea on every page has problems standing under it, marked with a reason rather than a tick, and any sentence you do not believe can be opened and argued with. None of that can happen on a page nobody owns.
The contents