Answers · How models work

What does it mean if two words are close together in embedding space?

In shortIt means training placed their rows in similar positions because the pieces appeared in similar company, so one can often stand in for the other in text; it is not a direct synonym score.

It means the training procedure placed the rows for those two pieces in similar positions because the pieces appeared in similar company. That makes them good substitutes in text, not necessarily synonyms, and the nearness is a learned pattern from the corpus rather than a direct statement about meaning.

What the number is

Nearness is normally read as cosine similarity, the cosine of the angle between the rows. The dot product alone would reward long rows, and a row grows longer when its piece is frequent, so the product of the two lengths is divided out. After that division only direction counts. A value near the top of the scale means the rows point almost the same way; a value near zero means they are close to perpendicular; a negative value means they point in opposing directions, which is rare for real pieces because opposites tend to share company rather than avoid it.

\operatorname{sim}(\mathbf{a}, \mathbf{b}) = \frac{\mathbf{a} \cdot \mathbf{b}}{\lVert \mathbf{a} \rVert \, \lVert \mathbf{b} \rVert}

Take two rows $\mathbf{a}=(3,4)$ and $\mathbf{b}=(4,3)$ . Their dot product is $3\cdot4+4\cdot3=12+12=24$ . The length of $\mathbf{a}$ is $\sqrt{3^2+4^2}=\sqrt{9+16}=\sqrt{25}=5$ , and the length of $\mathbf{b}$ is $\sqrt{4^2+3^2}=\sqrt{16+9}=\sqrt{25}=5$ . The cosine is therefore $24/(5\cdot5)=24/25=0.96$ . The rows point almost the same way. That calculation is complete as arithmetic and silent about meaning: it would give the same answer for two pieces that mean the same thing, for a spelling variant, and for a pair of opposites that the text used in the same slots.

Why the synonym reading fails

The natural guess is that a small angle is a synonym score. It is not. A row is adjusted when its piece appears and moving the row would improve the prediction of the surrounding pieces. The position a row settles into is therefore fixed by what tends to appear around its piece. Pieces that appear in similar company get pushed towards similar positions, because similar positions serve similar surroundings equally well. Hot and cold appear in almost the same company, as in the water is ..., it feels ..., a ... drink, so their rows end up close even though the words are opposites. Closeness records substitutability in the training text. That often coincides with similar meaning, and sometimes coincides with opposite meaning, and the table has no separate signal that tells those cases apart.

A high cosine therefore licenses a cautious sentence: one piece could often stand in for the other without the sentence becoming strange. It does not license the sentence that the pieces mean the same thing. Closeness also picks up spelling variants, grammatical variants, register variants, and pairs that are merely used in the same topic. No individual number in a row stands for an attribute either. Training has no reason to prefer one set of axes over another, and the same arrangement rotated as a whole serves equally well, so only relative positions carry information. Asking which coordinate means plural is like asking which direction in a room is north when the room could be rotated.

When a nearest-neighbour list is used in practice, the useful reading is comparative. Two rows close together say the corpus put their pieces in overlapping company, and the strength of that overlap has to be checked against other pairs in the same table. The same cosine means different things in different tables because the training text and the vocabulary differ. A value that is high in one table may be ordinary in another, so the absolute number is less informative than the ranking around it.

Where the account stops

This is the whole story only for one fixed row per entry. If a word has several senses, one row has to average them, so closeness to that row mixes the senses and can look misleadingly strong. The arithmetic on rows also works less well than demonstrations suggest. Some attributes form consistent directions, but the nearest row to a computed point is frequently one of the inputs, which is why demonstrations quietly exclude them, and the effect weakens sharply when the text marked an attribute inconsistently. Frequent pieces also win disproportionately because their rows sit in denser regions. Finally, the table is faithful to its corpus. Whatever associations the text contained, including unwanted ones, are in the arrangement because they were in the text. It reports those associations; it does not endorse them.

Worth keeping

Closeness in an embedding table means the training text used the two pieces in similar company, so they are often interchangeable in a sentence.
It is not a synonym score: opposites and spelling variants can sit close too.
Only relative positions carry information; no single coordinate in a row stands for an attribute.

Where this goes next

This question is one step of How Text Becomes Numbers, a course that works through its subject from the beginning. The page that teaches this part of it is Directions That Mean Something, and it reads the whole way through whether or not you have an account.

What an account changes is the practice: problems standing under every idea, marked with a reason rather than a tick, in a copy of the course that is yours.

Read that page The whole course