Two Obvious Vocabularies, Both of Which Fail

A model works on numbers. Text is not numbers. Something has to stand between

them, and that something is a numbered list of byte strings, together with a

rule for cutting any input into entries of the list.

That is the whole of it. The list is the vocabulary, usually a few tens of

thousands of entries, and the model only ever sees positions in the list. It

never sees letters. If two different inputs cut into the same sequence of

numbers, the model cannot tell them apart, and if the same word cuts into

different numbers in different places, the model has to learn both.

The obvious designs for that list are the two ends of a range: one entry per

word, or one entry per character. Both are tried, both fail, and the reasons

they fail are what the real answer is built out of.

Collect every distinct word in a large corpus, number them, and you have a

vocabulary. English has perhaps fifty thousand words in common use, which is a

convenient size, and each entry carries real meaning, which seems ideal.

Then the system meets a word that is not on the list. Not occasionally: this

happens on a sizeable share of real inputs, and it happens forever, because

language is an open set and a list is a closed one. Names are the obvious case,

and so are typos, compounds, product codes, slang coined last month, and the

same word with a suffix the corpus never showed.

There is no good move when the list misses. Mapping every unknown word to a

single entry meaning "something I do not know" throws away the one place the

information was: every name in the document becomes the same symbol, and the

model cannot tell one from another or copy it into its answer. Growing the list

does not end the problem, it only makes the tail thinner while making the output

layer, which has one entry per vocabulary item, proportionally more expensive.

And the list also encodes an assumption that words are separated by spaces,

which is true of English and false of Chinese, Japanese and Thai. For those

languages the question of what counts as a word is itself a research problem,

so the vocabulary cannot be built before it is answered.

Go to the other end. Use one entry per character, or better, one per byte. Now

the vocabulary has a couple of hundred entries, nothing is ever unknown, and

every language in the world is covered by the same list.

The cost is length. An English word is about five characters plus a space, so

the same page that is a thousand units long as words is around six thousand

units long as characters.

Six times longer is not a small inconvenience. Cost is charged per unit, so

every request costs six times as much. The context window is counted in units,

so a model that holds thirty pages of pieces holds five pages of characters. And

the work inside the model grows faster than linearly with sequence length, so

the training bill rises more than sixfold for the same text.

There is a subtler cost too. Relationships the model needs to learn are now

further apart. Two words that were adjacent are now twelve units apart, and a

pattern that spanned a paragraph now spans six times the distance, which is

measurably harder to learn.

Both failures are failures of a fixed granularity. The fix is to let the

granularity vary: frequent things get one entry, infrequent things get cut up.

This is the design every current model uses. The vocabulary holds whole words

for the common ones, fragments for the rest, and every single byte as a last

resort, which is what makes it total. Nothing is ever unknown, because the worst

case is that a string is spelled out one byte at a time.

The remaining question is which pieces. There are enormous numbers of possible

vocabularies of thirty thousand pieces, and the right one depends on the corpus:

a vocabulary fitted on English is a poor one for code and a terrible one for

Thai. That choice is made by an algorithm, run once, before training, and the

next lesson runs it by hand.