Four Machines, and Three of Them Are Waiting
Last timeCutting the Model Up
Lay the layers out in stages and each device waits for the one before it. The idle fraction follows from two numbers, and splitting the batch into pieces is what shrinks it.
The previous split cut every layer across devices and needed the fastest wire
in the building. This one cuts between layers instead, which produces a very
different communication pattern and a very different problem.
Stages
Take an eighty layer model and four devices. Give the first device layers one
to twenty, the second twenty-one to forty, and so on. Data enters at stage one,
flows through, and the loss is computed at stage four. The gradient then flows
back the other way.
The idle time
Now draw the timeline, and the problem appears immediately.
At the start of a step, stage one begins work. Stages two, three and four have
nothing to do, because nothing has reached them. Stage two starts when stage
one finishes, and stage four does not start until three stages have each
completed their work. At the end of the step the reverse happens: stage one
finishes its share and has nothing left while the others are still going.
With four stages, three units of time are wasted at the start and three at the
end. The devices are busy for part of the step and idle for the rest, and no
amount of hardware changes that: it is a property of the arrangement.
The lesson stops here
3 more paragraphs to go
You have read the opening. The rest of the argument, the problems that check whether it landed, and the lines worth keeping at the end all come with a plan.
The first lesson of every course in the library reads the whole way through, free, so you can see exactly what the rest of them are.
See the planThe contentsThis is the reading half
Starting the course gives you your own copy of it. Every idea on every page has problems standing under it, marked with a reason rather than a tick, and any sentence you do not believe can be opened and argued with. None of that can happen on a page nobody owns.
The contents