Why the Position Signal Is a Ladder of Sine Waves

Last timeWhy More Than One Head

Attention reads its input as a set, so order has to be added to the vectors first, and section 3.5 of the paper adds it as 256 sine and cosine pairs spanning four orders of magnitude.

Section 3.5 is half a page long and it is the part of the paper that aged

worst, which makes it the most instructive section in it. Everything the authors

do there is forced by a constraint they state in one sentence, and the specific

answer they chose has been replaced, so reading it carefully teaches both what

the problem is and how to tell a repair from the thing being repaired.

The constraint first, because the paper states it and does not prove it.

Nothing in the block knows where anything is

Take an attention block and delete the position signal, so the input is just the

token embeddings. Now permute the positions. Every query, key and value is

carried along with its own token, so every score in the block is still a score

between the same two tokens, the softmax over a permuted set of keys produces

the same weights attached to the same keys, and the weighted sum of values is

unchanged. The output at the new location of a token is what the output at its

old location used to be. The block is permutation equivariant: permuting the

input permutes the output and changes nothing else.

The rest of the layer cannot rescue it. The feed forward network is applied to

each position separately and identically, and so is the normalisation, so

neither of them can distinguish an ordering either. Stack six such layers and

the encoder computes a function of the multiset of tokens. Not approximately,

and not until it has learned better. The function has no argument in which the

order could be recorded.

So order has exactly two doors. It can be added to the vectors going into the

first layer, which is what this paper does and what the whole of section 3.5 is

about, or it can be put into the score, which is what nearly everything since

has done. One wrinkle is worth noting before it causes an argument later. The

decoder masks out the positions after the current one, and the mask is defined

by index and not by content, so a decoder is not permutation equivariant even

with the position signal removed. Some information about ordering reaches a

causal stack through the mask alone, which is why the question of position is

genuinely harder on the encoder side.

FIG 1The position encoding, exactly as section 3.5 gives it

: the position in the sequence, counting from zero
: the index of the coordinate pair, running from 0 to 255 at a width of 512
: the width of the stream, 512, so there are 512 coordinates to fill and 256 pairs
: the chosen spread of the ladder; the denominator runs from 1 at the first pair to nearly 10,000 at the last

Two coordinates at a time share one frequency, one taking its sine and the other its cosine. That pairing is not decoration and it is what the next section rests on. The vector produced is added to the token embedding, which is why it has to be the same width as the stream.

The ladder of frequencies

Write the frequency of pair $i$ as $\omega_i = 1 / 10000^{\,2i/512}$ , so that

the two coordinates of that pair are $\sin(\omega_i \cdot pos)$ and

$\cos(\omega_i \cdot pos)$ . The wavelength is $2\pi / \omega_i$ . At the first

pair, $i = 0$ , the exponent is zero, the denominator is 1 and the wavelength is

$2\pi \approx 6.28$ positions. The exponent reaches its largest value at

$i = 255$ , where $2i/512 = 510/512$ , so the denominator is

$10000^{510/512} \approx 9647$ and the wavelength is about 60,600 positions.

The ratio between neighbouring pairs is $10000^{2/512} \approx 1.0366$ , a

steady 3.66 per cent, applied 255 times.

That is worth lining up against what the paper says, which is that the

wavelengths form a geometric progression from $2\pi$ to $10000 \cdot 2\pi$ . The

upper end of that range needs $2i/d_{\mathrm{model}} = 1$ , which means

$i = 256$ , and there is no pair 256. The paper is naming where the progression

would arrive one step past its last term. The longest wavelength that exists in

a width-512 model is about $9647 \cdot 2\pi$ , not $10000 \cdot 2\pi$ , and the

difference matters not at all numerically and a great deal when you are trying

to reconstruct the indexing from the sentence.

The lesson stops here

5 more paragraphs to go

You have read the opening. The rest of the argument, the problems that check whether it landed, and the lines worth keeping at the end all come with a plan.

The first lesson of every course in the library reads the whole way through, free, so you can see exactly what the rest of them are.

See the plan The contents

This is the reading half

Starting the course gives you your own copy of it. Every idea on every page has problems standing under it, marked with a reason rather than a tick, and any sentence you do not believe can be opened and argued with. None of that can happen on a page nobody owns.

The contents

Why the Position Signal Is a Ladder of Sine Waves

Nothing in the block knows where anything is

The ladder of frequencies

5 more paragraphs to go

This is the reading half

Read alongside