Read the 2017 paper that introduced the transformer from its first table to its last, rebuilding every equation and every number in it yourself, and come out able to say which of its choices every model still makes and which were quietly abandoned

Reading the Paper That Started It

A guided read of the paper that replaced recurrence with attention, rebuilding its equations, its training recipe and its tables from scratch.

8 lessons, written and corrected before you arrived. Reading them here needs no account. The first reads the whole way through; the others open and then stop, because a page nobody owns cannot tell who is reading it. Starting the course gives you your own copy, where every idea has problems standing under it and you can ask about any sentence.

Start reading

01Why Attention Replaced Recurrence When It Costs MoreRebuilding the paper's opening comparison in numbers, where attention turns out to be the dearer layer and the deciding column is the one nobody quotes.
02What the Attention Equation Actually Computesopening onlyEquation one of the paper read matrix by matrix, with the square root in its denominator derived from the variance of a dot product and the mask put where the paper puts it.
03Why Eight Attention Heads and Not One Wide Oneopening onlyEight heads of sixty-four cost exactly what one head of five hundred and twelve costs, in weights and in multiplications, so the split has to be buying something else.
04Why the Position Signal Is a Ladder of Sine Wavesopening onlyAttention reads its input as a set, so order has to be added to the vectors first, and section 3.5 of the paper adds it as 256 sine and cosine pairs spanning four orders of magnitude.
05Most of the Block Is Not Attentionopening onlyTwo thirds of an encoder layer's weights sit in the wide inner layer rather than in attention, and the skip connection and the normalisation each answer a question attention does not.
06The Training Recipe, Read as a List of Repairsopening onlySection five is four decisions, each fixing something specific: a rate that starts near zero, targets that are never certain, batches counted in tokens, and the last five checkpoints averaged.
07Reading the Results Tables With the Cost Column Inopening onlyThe results table in the 2017 transformer paper has a training cost column, and rebuilding it from the paper's own footnote changes which row wins.
08Sorting the Paper's Choices Into Three Pilesopening onlyWhich of the 2017 decisions are still in the models people run, which were quietly replaced, and which were abandoned along with the reasons given for them.