Answers · How models work
What is wrong with just tokenizing text one character at a time?
In shortNothing is ever unknown, but a passage becomes roughly five times longer than it needs to be, and attention cost grows faster than length, so you pay several times over for the same text.
Nothing is unsafe about it. Every string is made of characters, so no input is ever unknown and the round trip back to text is exact. The problem is the other side of the trade: a passage becomes about five times longer than it needs to be, and the cost of running a model over it grows faster than the length does.
The count
Take a hundred words of ordinary English. An average word runs to about five characters once the space in front of it is counted, so cutting at every character produces five hundred pieces. Cutting at spaces produces a hundred. Learned pieces, where common words stay whole and rare ones break into fragments, land around one and a third pieces per word, so about a hundred and thirty.
Of those five hundred, a hundred are the spaces themselves, which leaves four hundred letters for a hundred words: four letters per word on average, with the space carried as a piece in its own right so the text can be rebuilt exactly. Five hundred pieces against a hundred and thirty is a factor of nearly four, for the same sentence.
| Scheme | List size | Pieces per word | Pieces for 100 words | Unknown inputs |
|---|---|---|---|---|
| Characters | a few hundred | 5 | 500 | never |
| Words | hundreds of thousands | 1 | 100 | common, and all collapse into one entry |
| Learned pieces | tens of thousands | about 1.3 | 130 | never |
Length alone understates it, because attention compares every position with every other. The number of such comparisons is the square of the sequence length, so five hundred pieces cost comparisons where a hundred and thirty cost . Nearly fifteen times the work for the same paragraph, and a factor of twenty-five against cutting at spaces.
The natural guess is that the smallest unit must be the safest, since a character can never be missing from the list. That reasoning is sound about coverage and silent about cost, and cost is where the scheme loses. It is also the wrong place to defend the property: fragments bottom out at single characters, so a piece-based vocabulary inherits the guarantee that nothing is unknown while giving back most of the length. You do not have to choose between the two.
There is a second bill that does not show up in the arithmetic. A model reading single characters has to learn, from data and using capacity it could spend elsewhere, that c, a and t in that order constitute a word. Word identity has to be reconstructed inside the network rather than arriving in the input. The same capacity spent on the actual task buys more.
Where it stops being true
The penalty is per unit of text, so it never reverses on long passages. It matters less when the sequences are short anyway, when the alphabet is small and regular, or when the text is so full of strings no vocabulary could hold that the fallback is doing all the work, as with noise, identifiers and code. Even then the length is still there; you are accepting it rather than escaping it. And cutting spaces as ordinary characters has a consequence people do not expect: the piece for a word after a space and the piece for the same word at the start of a line are different entries. A word typed without a leading space can behave differently from the identical word with one, because the model has seen the two far more often, one than the other.
Character cutting is the right answer to a different question, which is how to guarantee that no input is ever unrepresentable. It is a poor answer to how to spend a fixed budget of sequence length, and that is the question a model actually asks.
Worth keeping
- Cutting at every character never produces an unknown input, but it makes a passage roughly five times longer than cutting at spaces and about four times longer than learned pieces.
- Attention cost grows with the square of the sequence length, so a hundred-word passage at character level costs nearly fifteen times what the same passage costs with learned pieces.
- Learned pieces keep the guarantee that nothing is unknown, because fragments fall back to single characters, so the choice is not between coverage and length.
Where this goes next
This question is one step of How Text Becomes Numbers, a course that works through its subject from the beginning. The page that teaches this part of it is Where to Cut a Word, and it reads the whole way through whether or not you have an account.
What an account changes is the practice: problems standing under every idea, marked with a reason rather than a tick, in a copy of the course that is yours.
Read that pageThe whole courseOther questions people ask about this
- Why do models fail at counting letters or handling numbers and code?Models often fail at letters, digits and code because the text is chopped into pieces before the model sees it, hiding individual characters and place values.
- Why do tokenizers split words into smaller pieces instead of keeping whole words?Tokenisers split words into pieces because a fixed list of whole words is enormous and still cannot handle names, typos or new words, while common fragments can build any word without an unknown entry.
- What does it mean if two words are close together in embedding space?It means training placed their rows in similar positions because the pieces appeared in similar company, so one can often stand in for the other in text; it is not a direct synonym score.
- Why does the same word get a different vector depending on the sentence it appears in?The lookup gives the same word the same starting row every time; the different vector is built later by mixing in the words around it.
- Why does the model get stuck repeating the same sentence over and over?The model is not stuck in a memory loop: once it has written a phrase, that phrase is in its context and makes writing it again more likely, so a decoder that always picks the top word cannot leave.
- Why does diffusion add Gaussian noise instead of any random corruption?Diffusion adds Gaussian noise because Gaussian errors combine exactly: each step keeps the spread fixed and gives a closed-form noisy picture at any later step, which makes the reverse step learnable.