Reading the Paper That Started It
Sorting the Paper's Choices Into Three Piles
Last timeReading the Results Honestly
Which of the 2017 decisions are still in the models people run, which were quietly replaced, and which were abandoned along with the reasons given for them.
Nine years of models later, the useful question about this paper is not whether
it was right. It is which of its decisions are still in the thing you would
build today, which were swapped out so quietly that most readers never noticed,
and which were abandoned together with the arguments made for them. Those three
piles are not the same size, and sorting a decision into the right one needs a
test better than whether the word still appears in the code.
The test used here is this. A choice survived if a model built now would make
the same one and for the same reason. It was replaced if the job is still being
done but by different machinery. It was abandoned if nobody is doing the job at
all. The third pile is the interesting one, because a choice usually arrives
with a justification, and the two can be abandoned separately.
| Pile | What is in it |
|---|---|
| Survived | The scaling divisor, the per head width arithmetic, the residual plus normalise shape, the inner layer being wider than the stream |
| Replaced | The position signal, the place the normalisation sits, the inner layer's internal shape, the encoder and decoder pair |
| Abandoned | The extrapolation argument for sinusoids, restricted neighbourhood attention, checkpoint averaging, most of the regularisation |
The one line that nothing has replaced
Attention scores are dot products of a query and a key, divided by the square
root of the key width, before the softmax. The divisor is the piece most worth
understanding, because its survival is not luck.
An earlier reading of that equation derived the divisor from the paper's own
footnote, which supposes the entries of a query and a key to be independent,
with mean zero and variance one, and that derivation is not repeated here. What
it established is the fact needed now, the size of an unscaled score: the dot
product of two such vectors of width has variance , so its standard
deviation is , which at the paper's per head width of 64 is
. Scores of size 8, with the largest in a long row several times
that again, saturate a softmax and leave almost no gradient anywhere in it, and
dividing by 8 puts them back at size 1.
What is new is how little that argument rests on. It mentions the width of
a head and nothing else. It does not depend
on the task, the data, the depth of the stack, the position signal or the
activation in the inner layer, all of which have changed since. This is why the
same expression is in every architecture that followed, including the three cited
here that each replaced something else, and why the first pile has a line in it
that has not moved at all.
The head arithmetic survived with it. Eight heads at 64 apiece is 512, the model
width, and that is not a coincidence but the design rule the paper states, that
the per head width is the model width divided by the head count, chosen so that
the total cost matches single head attention at full width. A model built today
with 32 heads of 128 across a width of 4,096 is applying the same rule.
- the rescaling, sitting outside the addition, so the residual sum is rescaled before it travels on to the next layer
- the residual sum: the input kept as it was, plus whatever this sub-layer computed from it
The schedule is the receipt
The reason the placement changed is visible in the paper without any later
reference, in the learning rate schedule. An earlier reading took that schedule
as one of the four decisions in the recipe and worked out what its warming
constant sets; the same rule is read here as evidence about the structure, which
is a different question and gets a different answer.
What that earlier reading established is the one fact needed now: the rule is
the smaller of a rising line and a falling curve, the two cross exactly at the
warming step, and so the rate peaks at step 4,000 and falls from there. Nothing
about that derivation is repeated below. What is new is the size of the peak.
The lesson stops here
6 more paragraphs to go
You have read the opening. The rest of the argument, the problems that check whether it landed, and the lines worth keeping at the end all come with a plan.
The first lesson of every course in the library reads the whole way through, free, so you can see exactly what the rest of them are.
See the planThe contentsThis is the reading half
Starting the course gives you your own copy of it. Every idea on every page has problems standing under it, marked with a reason rather than a tick, and any sentence you do not believe can be opened and argued with. None of that can happen on a page nobody owns.
The contents