Reading the Paper That Started It
The Training Recipe, Read as a List of Repairs
Last timeThe Block Built Around Attention
Section five is four decisions, each fixing something specific: a rate that starts near zero, targets that are never certain, batches counted in tokens, and the last five checkpoints averaged.
Everyone has read section three. Section five is two thirds of a page and it is
the part that decides whether any of section three works at all. Read as a list
rather than as prose, it is four repairs, and each one is aimed at something
specific that goes wrong without it.
The scale first, because it is smaller than people remember. The base model was
trained on one machine with eight P100 processors, at about 0.4 seconds a step,
for 100,000 steps. That is 40,000 seconds, or 11.1 hours, against the roughly 12
hours the paper states, so the arithmetic closes. The big model ran at 1.0
second a step for 300,000 steps, which is 300,000 seconds, 83.3 hours, 3.47
days, against the 3.5 days stated. The data was about 4.5 million English and
German sentence pairs with a shared byte-pair vocabulary of about 37,000 tokens.
- the only place the width enters, dividing every rate in the schedule by the square root of 512, about 22.6
- the falling branch, which governs everything after the peak
- the rising branch, a straight line through the origin whose slope is set entirely by the warming constant
The rate that starts almost at zero
The two branches are equal when the step number raised to minus a half equals
the step number times the warming constant raised to minus three halves, and
rearranging gives the step equal to the warming constant. So the crossing is at
step 4,000 by construction, and the rising branch is lower before it and higher
after, which is why the minimum picks out exactly the shape wanted.
Now the step that reading the formula as written does not give you. Call the
crossing value the peak. On the rising branch the rate is the peak times the
step divided by 4,000, and on the falling branch it is the peak times the square
root of 4,000 divided by the step. Both lines follow by dividing the formula by
the peak and cancelling the width factor, and both are free of constants.
Two things fall out immediately. At step 1 the rate is the peak divided by
4,000, so the warming raises the rate by a factor of four thousand, linearly,
over a stretch that is 1,600 seconds of a twelve hour run, four per cent of it.
And the peak itself is the width to the minus a half times the warming constant
to the minus a half, so the warming constant sets the height of the peak as well
as its position. Doubling it to 8,000 does not give the model twice as long to
settle at the same peak; it divides the peak by the square root of two. That is
the single most commonly missed property of this schedule.
The lesson stops here
7 more paragraphs to go
You have read the opening. The rest of the argument, the problems that check whether it landed, and the lines worth keeping at the end all come with a plan.
The first lesson of every course in the library reads the whole way through, free, so you can see exactly what the rest of them are.
See the planThe contentsThis is the reading half
Starting the course gives you your own copy of it. Every idea on every page has problems standing under it, marked with a reason rather than a tick, and any sentence you do not believe can be opened and argued with. None of that can happen on a page nobody owns.
The contents