Where the Square Root of $d_k$ Comes From

Last timeQuery, Key and Value

The division by the square root of the key dimension is not a tuning constant. It falls out of the variance of a dot product between independent vectors.

Of all the pieces of the attention formula, the $\sqrt{d_k}$ in the denominator

looks most like something someone tried until training stopped diverging. It is

not. It is the standard deviation of the quantity above it, and the derivation

takes about five lines.

The setup

Assume, for the length of this argument, that the components of a query vector

$q$ and a key vector $k$ are independent random variables with mean zero and

variance one. This is roughly what a freshly initialised network gives you, and

roughly what normalisation maintains during training.

A single attention score is their dot product:

q \cdot k = \sum_{i=1}^{d_k} q_i k_i

The question is how large this number typically is, and specifically how its

size depends on $d_k$ .

The lesson stops here

11 more paragraphs to go

You have read the opening. The rest of the argument, the problems that check whether it landed, and the lines worth keeping at the end all come with a plan.

The first lesson of every course in the library reads the whole way through, free, so you can see exactly what the rest of them are.

See the plan The contents

This is the reading half

Starting the course gives you your own copy of it. Every idea on every page has problems standing under it, marked with a reason rather than a tick, and any sentence you do not believe can be opened and argued with. None of that can happen on a page nobody owns.

The contents

Where the Square Root of dkd_kdk​ Comes From

The setup

11 more paragraphs to go

This is the reading half

Read alongside

Where the Square Root of $d_k$ Comes From