Answers · How models work

What do multiple attention heads actually give you that one big head doesn't?

In shortSeveral heads let a model track several relationships at once, each in its own lower-dimensional subspace, instead of squeezing every relationship into one weighted average that destroys information.

Multiple heads let the model keep several different weighted averages at once, each in its own low-dimensional subspace, instead of forcing one average to carry every relationship. A single head produces one mixture per token, and a mixture destroys information: once two value vectors are blended, nothing downstream can separate them again.

\mathrm{head}(X)_t = \sum_{s} \alpha_{t s} \, (X W_V)_s, \qquad \sum_s \alpha_{t s} = 1

One average cannot hold two answers

Take a sentence where one token needs two different things at once: 'The researcher who wrote the paper said it was wrong.' For the token 'said', a useful representation depends on its subject 'researcher' (four tokens back, across a relative clause) and on its complement clause beginning at 'it'. These are different questions with different answers. One head must answer both with one set of weights summing to one. If it puts 0.45 on 'researcher', 0.40 on the complement, and 0.15 elsewhere, the output is 0.45 v_researcher + 0.40 v_complement + 0.15 v_other, a blend of two unrelated vectors. Nothing downstream can undo that blend. Two heads do not split weight: head A puts 0.95 on 'researcher', head B puts 0.90 on the complement, and both vectors survive into the output because they occupy different coordinates.

FIG 1One head forced to blend two relationships

The weights sum to one, so attending to two relationships means splitting weight and producing a blend that represents neither cleanly.

The construction

Run $h$ heads in parallel. Each head has its own projections $W_Q^i$ , $W_K^i$ , and $W_V^i$ , so each computes its own scores and its own weighted average. The $h$ outputs, each of width $d_v$ , are concatenated and multiplied by one output matrix $W_O$ .

\mathrm{MultiHead}(X) = \mathrm{Concat}(\mathrm{head}_1, \dots, \mathrm{head}_h)\, W_O, \qquad \mathrm{head}_i = \mathrm{Attention}(X W_Q^i, X W_K^i, X W_V^i)

Why this costs nothing

The obvious objection is that eight heads should cost eight times as much. They do not, because head width shrinks as head count grows: $d_k = d_{model} / h$ . The attention cost per head has two parts. The score matrix is $L \times L$ and each entry is a dot product over $d_k$ terms, giving $L^2 d_k$ . The weighted sum of values is the same order. Multiply by $h$ heads: $h \cdot L^2 d_k = h \cdot L^2 \cdot (d_{model}/h) = L^2 d_{model}$ . The head count cancels. Eight heads of width 64 cost $L^2 \cdot 512$ , exactly what one head of width 512 costs.

FIG 2

Configuration	Heads	Width each
Single wide head	1	512
Original transformer	8	64
Many narrow heads	32	16

What the output projection is for

Concatenation alone would put head one in coordinates 1 through 64, head two in 65 through 128, and so on. Each head would write into a fixed, private slice, and nothing downstream could receive contributions from two heads in the same coordinate. $W_O$ is a full matrix mapping the concatenated vector back to model width. Each head's output can be written into any direction. Two heads can add into the same direction; one head's output can be spread across many. This is the mechanism by which a head in one layer can write something that a head in a later layer reads.

The natural guess is that heads are just the same attention run several times for redundancy. That fails because each head has its own projections $W_Q^i$ , $W_K^i$ , $W_V^i$ . Each head computes a different similarity matrix $W_Q^i (W_K^i)^\top$ and a different weighted average in a subspace of dimension $d_k = d_{model}/h$ . They are not copies; they are different questions asked of the same sequence. The redundancy that does exist is a by-product of training, not the design.

It stops being true that every head does something useful. Probing trained models finds that a minority of heads have clean, describable jobs, such as attending to the previous token or tracking a syntactic dependency, and a substantial fraction can be removed with little effect on the output. Head count is chosen with slack, not tuned to a minimum. Also, narrower heads mean simpler individual similarity functions: the learned matrix $W_Q W_K^\top$ has rank at most $d_k$ , so splitting into more heads trades expressiveness per head for coverage across heads.

Worth keeping

A single attention head produces one weighted average per token, and averaging is irreversible, so any relationship not weighted highly is lost.
Splitting the model width into h heads of width d_model/h costs the same as one wide head, because the per-head cost falls in proportion to the head count.
The output projection lets head outputs mix into any direction, so heads are not locked into fixed, disjoint slices of the representation.

Where this goes next

This question is one step of The Mathematics of Attention, a course that works through its subject from the beginning. The page that teaches this part of it is Multiple Heads, and it reads the whole way through whether or not you have an account.

What an account changes is the practice: problems standing under every idea, marked with a reason rather than a tick, in a copy of the course that is yours.

Read that page The whole course