Why Attention Needs Three Matrices, Not One

Attention is usually introduced with its final formula, and the three matrices

$W_Q$, $W_K$ and $W_V$ arrive already named, as though the names explained

themselves. They do not. The useful question is what goes wrong without them,

and the answer is specific enough to derive.

Start with what you have. A sequence of $L$ tokens, each already a vector of

width $d_{model}$, stacked into a matrix $X$ of shape $(L, d_{model})$.

You want each token to gather information from the others, weighted by how

relevant they are. Relevance between two vectors has an obvious candidate: the

dot product. Large when they point the same way, near zero when they are

unrelated. So the simplest possible mechanism is to score every pair directly

and use those scores to mix the tokens.

This runs. It produces output of the right shape. It is also broken in three

separate ways, and each of the three matrices exists to fix exactly one of them.

$S = XX^\top$ means $S_{ij} = x_i \cdot x_j$, and the dot product does not care

about order. So $S_{ij} = S_{ji}$, always, as an algebraic fact rather than a

tendency. Any matrix of the form $MM^\top$ is symmetric by construction.

That forces every attention relationship to be mutual. How much the word it

attends to the cat must equal how much the cat attends to it.

Almost nothing in language works that way. A pronoun needs its antecedent

badly; the antecedent has little use for the pronoun. A verb needs its subject;

the subject is not especially interested in the verb. An adjective modifies a

noun in one direction only. Relevance is directional, and a symmetric score

matrix cannot represent direction at all.

Two separate projections fix this. Send the input through one matrix to ask a

question and through a different matrix to answer it:

Now $S_{ij} = (x_i W_Q) \cdot (x_j W_K)$, and swapping $i$ and $j$ gives

$(x_j W_Q) \cdot (x_i W_K)$, which is a different number whenever

$W_Q \neq W_K$. The constraint is gone. The model can learn that it should

look hard at cat while cat largely ignores it.

The names follow from the roles rather than the other way round. The query is

what a token is looking for. The key is what a token advertises about itself.

They are separate because asking and being asked are separate.

The parameter-free version has no way to improve. $XX^\top$ is a fixed function

of the input. If the useful notion of relevance for a task is not raw vector

similarity, and it usually is not, there is no mechanism by which the model

could discover that.

This is the deeper reason for the projections, and it is easy to miss behind

the symmetry argument. The dot product is fixed. The softmax is fixed. Neither

has a single parameter. **Everything an attention head knows is stored in its

projection matrices.** They are the entire learnable content of the mechanism.

What they learn is a subspace to compare in. $W_Q$ and $W_K$ together decide

which directions of the input space count toward relevance and which are

ignored, because the score can be rewritten to make the pairing explicit:

Written this way the naive attempt is revealed as a particular, and arbitrary,

choice: the one where the middle matrix is $I$. There is no reason the identity

should be the right similarity measure for a language task, and the projections

are simply the machinery for learning something better.

The third flaw is the least obvious and the most interesting.

In the naive version, the same vector $x_j$ does two jobs. It is matched

against (it determines whether token $j$ gets attended to) and it is passed on

(it is what token $i$ receives). Those are different jobs with different

requirements, and one vector cannot be optimised for both.

Consider a token whose useful contribution is a piece of factual content, but

which is found by matching on grammatical role. The features that make it

findable and the features that make it worth having need not overlap at

all. Forcing them into one vector means every improvement in retrievability

costs something in content, and the reverse.

The value projection separates them:

Now $W_Q$ and $W_K$ shape what makes a token findable, while $W_V$ shapes what

a token contributes once found. They can be optimised independently, because

they are.

The dictionary analogy usually offered for attention is worth stating carefully,

because the loose version obscures the point. In a dictionary you look up a

query, compare it against keys, and receive the associated value. The part worth

noticing is that the key and the value are stored separately. A dictionary in

which the key and the value were forced to be the same object would be nearly

useless, and that is precisely what the parameter-free version is.

One detail that looks arbitrary in the original paper: with a model width of

512 and eight heads, each head uses $d_k = 64$. The queries and keys are

projected into a space eight times narrower than the input.

This is not a compromise forced by memory. It follows from where those vectors

are used. Queries and keys appear only inside a dot product, and that dot

product collapses them to a single number. Their width therefore sets the rank

of the learned middle matrix $W_Q W_K^\top$, which is at most $d_k$, and

nothing else. No downstream component ever sees a query or a key.

The value projection is different. Values are summed and passed forward, so

$d_v$ has to line up with what the rest of the block expects. In practice the

two are set equal for convenience, but they are constrained by different things,

and only one of them is free.

Three matrices, three distinct failures repaired. Splitting query from key makes

relevance directional and makes it learnable. Splitting value from both means a

token's findability and its contribution stop competing for the same

coordinates.

The next lesson takes the one remaining unexplained term in the formula, the

division by $\sqrt{d_k}$, and shows that it falls straight out of the variance

of a dot product between independent vectors.