Answers · How models work
Why can't a language model just treat each whole word as one token?
In shortBecause a fixed list of words cannot contain a word nobody has written yet, and enlarging the list never catches up, while the model still has to put a number to every input it is given.
A whole-word vocabulary fails because a list is finite and language is not. Some word in the next document is not on it, and there is no sensible thing to do with that word: every unseen string has to collapse into the same placeholder, or the list has to grow, and growing it never closes the gap. The other end of the range, one entry per byte, never misses anything and pays for that by making the same text about six times longer.
The list that runs out
Fifty thousand entries covers the common words of English and looks like a reasonable vocabulary. Each entry means something, and a page of a thousand words becomes a thousand units. Take a fresh page of a thousand words and around forty-eight of them are not on the list, which is 4.8 per cent, roughly one word in twenty-one. That rate does not fall to zero as the corpus grows, because new names, product codes, compounds and spellings appear as fast as old ones are collected.
There is no good move for the miss. Give every unseen word one shared entry and the model cannot tell a place name from a surname from a typo, and it cannot copy either of them into its answer: the one place the information sat is exactly where the vocabulary is blank. Enlarge the list instead and the rate falls quickly and then crawls. Going from fifty thousand entries to a hundred thousand roughly halves a share that was already small, and it doubles the rows of the output layer, which carries one row per entry. Twice the parameters in that layer, for a problem that is still there.
| entries in the list | units for the page | unknown words per thousand |
|---|---|---|
| 50000 | 1000 | 48 |
| 256 | 6200 | 0 |
| 32000 | 1310 | 0 |
One token per word feels right because a word is the smallest unit that carries meaning, so every number would mean something and the sequence would stay short. Both halves of that fail at once. The words carrying the specific information in a sentence, the name, the typo, the invented compound, are precisely the rare ones, so the list is blank where the content is. And meaning is not preserved anyway: as separate entries, a stem and its inflections share nothing, so the model has to learn the same pattern once per form and gets no help from the ending.
What the other end costs
One entry per byte never misses. The sentence "the unhappiest wombat in Ulaanbaatar" is five words, forty-three bytes and twelve pieces. Forty-three over five is 8.6, so the byte version is nearly nine times longer than the word version; twelve over five is 2.4, so the piece version is under two and a half times longer and loses nothing.
Six times longer is charged six times over. Cost is billed per unit, so a request against a byte vocabulary costs about 6.2 times what the same request costs against a word vocabulary. The context window is counted in units, so a model that holds thirty pages of words holds five pages of bytes. And the work inside the model grows faster than linearly in sequence length, so the training bill for identical text rises by more than six times. Relationships also stretch: two words that were neighbours are now a dozen positions apart, and a pattern spanning a paragraph spans six times the distance, which is measurably harder to learn.
The design that survives varies the granularity instead of fixing it. Frequent strings get one entry, infrequent ones get cut up, and single bytes are kept as a last resort, which is what makes the list total. That gives 1310 units for the thousand-word page, about 1.31 units per word, and nothing is ever unknown: the worst case is that a string is spelled out one byte at a time.
The pieces are fitted to a corpus rather than to a language, so thirty-two thousand pieces chosen from English are a poor fit for code and a worse one for Thai, and a model serving many languages pays for fragments it rarely uses. A word list also assumes spaces separate words, which is true of English and false of Chinese, Japanese and Thai, where deciding what counts as a word is itself an open question, so the vocabulary cannot even be built before that question is settled.
Worth keeping
- A finite word list cannot cover an open language, so some input always has no entry, and no amount of enlarging the list removes that.
- A byte vocabulary never misses and makes each page about 6.2 times longer, which is charged per unit on every request and eats the context window.
- Keeping frequent strings whole and splitting the rest gives total coverage at about 1.31 units per word instead of 6.2.
Where this goes next
This question is one step of Cutting Text Into Pieces, a course that works through its subject from the beginning. The page that teaches this part of it is Why Not Just Words, and it reads the whole way through whether or not you have an account.
What an account changes is the practice: problems standing under every idea, marked with a reason rather than a tick, in a copy of the course that is yours.
Read that pageThe whole courseOther questions people ask about this
- Why does the same request cost more tokens in some languages than in others?A token is learned from a corpus, so a language that was rare in the training text is split into many more pieces for the same meaning, and the price multiplier follows.
- Why do tokenizers work on bytes instead of on characters?Tokenizers use bytes because 256 byte values can represent any text, while Unicode has too many characters to include safely.
- Why is 'hello' a different token from ' hello' with a space in front?A space is a byte that gets merged onto the front of the next word, so the tokenizer stores ' hello' and 'hello' as separate symbols with no shared structure.
- Why can't you just add new tokens to a tokenizer after training?Because each token id is tied to weights the model has already trained, so changing the vocabulary makes old numbers name different pieces while the model keeps the old meanings.
- Why does the model get stuck repeating the same sentence over and over?The model is not stuck in a memory loop: once it has written a phrase, that phrase is in its context and makes writing it again more likely, so a decoder that always picks the top word cannot leave.
- Why does diffusion add Gaussian noise instead of any random corruption?Diffusion adds Gaussian noise because Gaussian errors combine exactly: each step keeps the spread fixed and gives a closed-form noisy picture at any later step, which makes the reverse step learnable.