The Four Hops Between Storage and the Step
Last timeThe Loop Everything Sits In
A batch travels from a store, through host memory, through tokenisation, onto the device. Each hop can be the one the whole run is waiting on.
Where a batch comes from
The loop asks for a batch and gets one. That request is satisfied by a chain of
four hops, and the chain is almost always longer and slower than the step that
consumes it.
The first hop is the read. The data is sitting somewhere that is not the
machine: an object store, a network filesystem, a set of files on a local disk
if you are lucky. Bytes have to arrive. The second hop is the decode: those
bytes are in some container format, compressed, and have to be turned back into
records. The third hop is tokenisation, which turns a record of text into a list
of integers. The fourth hop is the transfer onto the device, across a bus, into
memory the chip can actually reach.
Only after all four does the loop have something to multiply.
The honest way to think about this is that a training run is a processor program
that occasionally asks an accelerator to do some arithmetic. People picture it
the other way round, and that is why pipeline problems are so often mistaken for
model problems.
Which side is waiting
There are only two states a run can be in. Either the chip has work and the
data pipeline is scrambling to keep up, or the pipeline has batches queued and
the chip is the bottleneck. The second is what you are paying for. The first is
you paying for an idle chip.
The number that settles it is the fraction of wall-clock time the chip spent
computing.
The lesson stops here
4 more paragraphs to go
You have read the opening. The rest of the argument, the problems that check whether it landed, and the lines worth keeping at the end all come with a plan.
The first lesson of every course in the library reads the whole way through, free, so you can see exactly what the rest of them are.
See the planThe contentsThis is the reading half
Starting the course gives you your own copy of it. Every idea on every page has problems standing under it, marked with a reason rather than a tick, and any sentence you do not believe can be opened and argued with. None of that can happen on a page nobody owns.
The contents