Answers · How models work
Why do tokenizers split words into smaller pieces instead of keeping whole words?
In shortTokenisers split words into pieces because a fixed list of whole words is enormous and still cannot handle names, typos or new words, while common fragments can build any word without an unknown entry.
Tokenisers split words into smaller pieces because a fixed list of whole words is enormous and still fails on anything it has not seen, while a list of common fragments can build any string from known parts. That keeps ordinary text short and makes rare or new words representable without a catch-all entry.
- the number of words in the text
- the average number of pieces per word
- the sequence length handed to the model
The length lever
For a passage of words, the sequence length is . With whole words, , so . With single characters, an average English word is around five characters including the space, so and . A learned piece scheme lands between those, often near for ordinary English, giving . Attention inside a model scales worse than linearly with sequence length, so the factor of five from characters is more than five times the cost, not merely a longer list.
Why the word list fails
The natural guess is that whole words are simpler because they carry meaning. The problem is coverage: names are effectively unlimited, typos are unlimited, and new words appear constantly. A word-level vocabulary that covers ordinary English has to be far larger than a character list, and it still needs one catch-all row for anything absent. Extending the list fixes one word at a time and never finishes, while the next unusual word has the same problem.
That catch-all entry is the part the natural guess misses. Two different names that both fall off a word list become the same input, indistinguishable in every layer that follows. A system built this way cannot tell one unknown person from another, and the failure is silent. Adding one name to the list fixes that name only; the next unusual string is back in the same catch-all. A piece scheme does not need to add anything, because it falls back on fragments that were chosen for being common.
The compromise is to keep frequent words whole and break rare ones into fragments that are themselves common. "help", "helpful" and "unhelpful" can share the piece for "help"; "token", "tokeniser" and "tokenisation" can share "token". A frequent word arrives as one piece, so ordinary text stays close to the word-level length. A rare word arrives as several pieces, and because the fragments bottom out at single characters, the scheme can represent any string. That is why the two failures disappear together: the sequence stays short where it matters and nothing is left unknown.
| scheme | p | L for W=100 | unknown input |
|---|---|---|---|
| characters | 5 | 500 | impossible |
| words | 1 | 100 | catch-all |
| learned pieces | 1.3 | 130 | impossible |
Read the middle column: characters make the sequence five times longer than words, since . Learned pieces make it only about thirty percent longer, since . Read the last column: only the word scheme has a catch-all, and that is the entry that loses information. The piece scheme is not the shortest possible sequence, but it is the shortest one that never produces an unknown input.
What the compromise costs
The cost is that word boundaries no longer line up with anything the model sees. A word may be one piece or several, and which it is depends on how often it appeared in the corpus used to build the vocabulary rather than on anything about the word itself. Two closely related words can be cut completely differently and share no pieces, so the model has to learn the relation from data. The space before a word is usually part of the piece, not a separator. That is why the same word at the start of a line and in the middle of a sentence are different entries, and why a word typed without a leading space can be cut differently. None of this is a bug: the round trip has to keep the space to be exact.
This also explains an oddity that looks like a tokeniser bug and is not. A word after a bracket or at the start of an input can be cut differently from the same word after a space, because the leading-space version is a different piece in the list. The model has seen one version far more often than the other, so the two can behave differently. The difference is in the input pieces, not in the word.
The scheme also stops being cheap when the vocabulary was built for a different language. On the language it was built from, a hundred words may produce about a hundred and thirty pieces. If the average rises to , the same hundred words produce three hundred pieces, so the sequence is more than twice as long, since is about 2.3, and the cost per sentence rises with it. That is a coverage question as much as a fairness one: the pieces exist, but they are not the pieces that language needs most often.
Worth keeping
- A whole-word vocabulary is large and still has a catch-all entry, so two different unseen words become the same input.
- A piece vocabulary can represent any string from known fragments, so nothing is ever unknown.
- The price of pieces is that word boundaries no longer align with what the model sees.
Where this goes next
This question is one step of How Text Becomes Numbers, a course that works through its subject from the beginning. The page that teaches this part of it is Where to Cut a Word, and it reads the whole way through whether or not you have an account.
What an account changes is the practice: problems standing under every idea, marked with a reason rather than a tick, in a copy of the course that is yours.
Read that pageThe whole courseOther questions people ask about this
- What is wrong with just tokenizing text one character at a time?Nothing is ever unknown, but a passage becomes roughly five times longer than it needs to be, and attention cost grows faster than length, so you pay several times over for the same text.
- Why do models fail at counting letters or handling numbers and code?Models often fail at letters, digits and code because the text is chopped into pieces before the model sees it, hiding individual characters and place values.
- What does it mean if two words are close together in embedding space?It means training placed their rows in similar positions because the pieces appeared in similar company, so one can often stand in for the other in text; it is not a direct synonym score.
- Why does the same word get a different vector depending on the sentence it appears in?The lookup gives the same word the same starting row every time; the different vector is built later by mixing in the words around it.
- Why does the model get stuck repeating the same sentence over and over?The model is not stuck in a memory loop: once it has written a phrase, that phrase is in its context and makes writing it again more likely, so a decoder that always picks the top word cannot leave.
- Why does diffusion add Gaussian noise instead of any random corruption?Diffusion adds Gaussian noise because Gaussian errors combine exactly: each step keeps the spread fixed and gives a closed-form noisy picture at any later step, which makes the reverse step learnable.