Most of the Block Is Not Attention

Last timeWhere Position Comes From

Two thirds of an encoder layer's weights sit in the wide inner layer rather than in attention, and the skip connection and the normalisation each answer a question attention does not.

Attention gets two and a half pages of the paper. The rest of the block gets one

short subsection and nine lines of section 3.1, and that proportion is almost

exactly inverted from where the weights are. In the base configuration an

encoder layer holds 3,145,728 weights, and 2,097,152 of them belong to the

sub-layer that never looks at any position but its own.

One layer, as the paper writes it

Section 3.1 gives the whole construction in a sentence. The encoder is a stack

of six identical layers. Each layer has two sub-layers, multi-head self

attention and a position-wise feed-forward network. Around each sub-layer there

is a residual connection followed by layer normalisation, so what a sub-layer

contributes is the normalisation of the input plus the sub-layer output. The

decoder is the same with a third sub-layer wedged in the middle, attending over

the encoder's output.

FIG 1One encoder layer, with the wrapper drawn in

Dropout is part of the wrapper rather than a stage of its own, and where it sits is worth a second look: it is on the sub-layer output, before the addition, so the thing dropped is always the contribution and never the stream it is added to. Interrupting that stream is the one thing the whole arrangement exists to avoid.

Section 5.4 is where the dropout comes from. It is applied to the output of each

sub-layer before that output is added and normalised, and also to the sum of the

embeddings and the position signal, at a rate of 0.1 in the base configuration.

The lesson stops here

8 more paragraphs to go

You have read the opening. The rest of the argument, the problems that check whether it landed, and the lines worth keeping at the end all come with a plan.

The first lesson of every course in the library reads the whole way through, free, so you can see exactly what the rest of them are.

See the plan The contents

This is the reading half

Starting the course gives you your own copy of it. Every idea on every page has problems standing under it, marked with a reason rather than a tick, and any sentence you do not believe can be opened and argued with. None of that can happen on a page nobody owns.

The contents

Most of the Block Is Not Attention

One layer, as the paper writes it

8 more paragraphs to go

This is the reading half

Read alongside