Why does attention need positional encoding if it already looks at the whole sequence at once?

Attention sees the whole sequence at once, but it sees it as a set: every score is a dot product and every output is a weighted sum, and neither operation refers to an index. Strip out position and "dog bites man" and "man bites dog" give the same three representations, merely placed in whichever slots the words occupy, so order has to be injected into the vectors themselves.

Write the token vectors as rows of $X$. The score from token $i$ to token $j$ is $q_i \cdot k_j = \sum_d q_{i,d} k_{j,d}$; it uses the two vectors and nothing else. Softmax normalises those scores across $j$, and a sum does not care in what order its terms are added. The output for token $i$ is then $\sum_j a_{ij} v_j$, another sum. Reorder the rows of $X$ by a permutation $P$ and the scores are the same numbers with rows and columns relabelled, the same softmax weights follow, and the outputs are the original outputs with rows relabelled.

Take the two orders in the table. If the three content vectors are $d$ for dog, $b$ for bites and $m$ for man, bare attention returns $[out_d, out_b, out_m]$ for order A and $[out_m, out_b, out_d]$ for order B. The vector attached to dog is identical in both, so nothing downstream can tell whether dog is the biter or the bitten. The three vectors are not scrambled; they are shuffled exactly as the input was. That is what permutation equivariance means, and it is why the mechanism cannot tell a sentence from an anagram of it.

Sinusoidal encodings add a vector of fixed sines and cosines to each token embedding. Every component lies in $[-1, 1]$, so position cannot dominate content, and the construction is defined for every integer position, so no maximum length is baked in. The property that earns its place is this: for any fixed offset $k$ there is a matrix $M_k$, depending on $k$ alone, such that $PE_{pos+k} = M_k PE_{pos}$. For one frequency pair this is the angle addition formula written as a rotation. Take frequency $\omega = 0.1$ radians per position and offset $k = 3$; then $\omega k = 0.3$ radians, with $\cos(0.3) \approx 0.955$ and $\sin(0.3) \approx 0.296$. Moving three positions forward rotates that pair by a fixed $0.3$ radians, the same rotation everywhere.

Rotary encodings apply the rotation later, to the query and key vectors themselves, rather than adding anything to the input. Rotations preserve lengths and angles under a shared rotation, and the score between positions $m$ and $n$ becomes $\langle R_m q,\; R_n k \rangle = \langle q,\; R_{n-m} k \rangle$. Absolute position has cancelled: the score for two tokens five apart is computed the same way whether they sit at positions 3 and 8 or at positions 3003 and 3008, since $8-3=5$ and $3008-3003=5$. That is why relative distance is built in rather than learned separately at every absolute position. It does not make length generalisation free. Models trained on short contexts still degrade on much longer ones, and the usual fixes rescale or interpolate the rotation frequencies; the difference is that the failure is a matter of degree, not the complete absence of a representation for unseen positions.