Derive the attention mechanism rather than describe it, well enough to read the equations in a paper without skipping them

The Mathematics of Attention

A derivation-first walk through the attention block, taking each design choice in turn and showing what breaks without it.

8 lessons, written and corrected before you arrived. Reading them here needs no account. The first reads the whole way through; the others open and then stop. Starting the course gives you your own copy, where every idea has problems standing under it and you can ask about any sentence.

Start reading

01Why Attention Needs Three Matrices, Not OneAttention projects each token into query, key and value spaces. Here is what breaks if you compare token vectors directly instead.
02Where the Square Root of $d_k$ Comes Fromopening onlyThe division by the square root of the key dimension is not a tuning constant. It falls out of the variance of a dot product between independent vectors.
03Why Softmax and Not Some Other Normalisationopening onlyScores have to become weights, and many functions could do that. Softmax is chosen for properties that are specific and checkable, not for convention.
04One Weighted Average Is Not Enoughopening onlyA single attention head produces one mixture per token, and a mixture destroys information. Heads are how the mechanism keeps several answers at once.
05Attention Cannot See Order, So Order Has to Be Addedopening onlyShuffle the input tokens and attention produces the same outputs, shuffled. Position must be injected into the vectors themselves, and how it is done constrains what the model can generalise to.
06The Skip Connection Is the Main Roadopening onlyThe residual path is usually described as a fix for vanishing gradients. In a transformer it is better understood as a shared channel that every block reads from and adds to.
07Why Transformers Normalise Across Features and Not Across the Batchopening onlyBatch normalisation works well in vision and badly in language. The reason is about which axis the statistics are taken over, and it decides more than stability.
08The Half of the Transformer That Does Not Move Informationopening onlyAttention moves information between positions but computes almost nothing nonlinear. The feed-forward block does the opposite, and holds most of the parameters.