Answers · How models work
Why do tokenizers work on bytes instead of on characters?
In shortTokenizers use bytes because 256 byte values can represent any text, while Unicode has too many characters to include safely.
A tokenizer starts from bytes because there are only 256 byte values, and every possible string is a sequence of them. Characters are not a closed alphabet: Unicode has nearly 150,000 of them and adds more, so a character-level vocabulary is either too large to afford or incomplete.
The alphabet problem
Take a 50,000-entry vocabulary. If one entry had to be a fallback for every Unicode character, 150,000 characters would need 150,000 / 50,000 = 3 times the slots, before a single word or subword had been learned. Cut the vocabulary to the characters actually seen in the corpus and the guarantee disappears: the first document containing an unseen script has no representation. Reserving one slot per character is therefore not a conservative choice; it is an impossible budget, and the alternative of ignoring unseen characters is a silent corruption. The only way out is an alphabet small enough to fit whole and complete enough that nothing can fall outside it.
Bytes are the smaller complete set
There are exactly 256 byte values. Put all 256 in the vocabulary and every file, every message, every malformed string and every binary blob has a representation, because that is what a file is: a sequence of bytes. The worst case is that the tokenizer spells a string out one byte per piece. That is long, but it is never impossible, and the merging algorithm does not change. It learns the same merges by counting pairs; it simply begins with 256 symbols rather than with the characters that happened to appear in the training corpus.
- bytes per character, fixed by the encoding: one for English, two for Cyrillic, three for Devanagari
- bytes per learned piece, fixed by the corpus: about four for English, barely above one for a script the corpus hardly contained
Take English as and suppose the learned vocabulary reaches bytes per piece. Then pieces per character. Take a script encoded at three bytes per character with almost no learned merges, so . Then pieces per character. The ratio is : the same meaning costs eight times as many pieces, and the only inputs to that number are the encoding's bytes per character and the corpus's merges per piece.
| script or encoding | b, bytes per character | s, bytes per piece | p = b/s, pieces per character | cost against English |
|---|---|---|---|---|
| English | 1 | 4 | 0.25 | 1 |
| Cyrillic | 2 | 1.5 | 1.33 | 5.33 |
| Devanagari | 3 | 1.5 | 2 | 8 |
The natural guess is that a character is a universal atom, the smallest thing a person can see in a string. That guess fails twice. First, a visible character is not one code point in Unicode: an accented letter can be a single code point or a plain letter followed by a combining accent, and several spaces, dashes and quotation marks have multiple code points that render identically. A byte-level tokenizer sees different byte sequences for the same visible string, cuts them into different pieces, and splits the model's evidence across two representations. Second, characters are not a small closed set. A character vocabulary that omits a script does not degrade gracefully; it produces an unrepresentable input. Bytes make the fallback total rather than approximate.
The guarantee is not free, and it is not evenly distributed. UTF-8 spends one byte on English letters, two on accented Latin, Greek, Cyrillic, Hebrew and Arabic, three on Devanagari, Chinese, Japanese, Korean and Thai, and four on emoji. That is the in the ratio. The other factor, , is set by the corpus: merges are learned by counting, so a script that is a small fraction of the corpus tends to get a similar fraction of the merges. A script at three bytes per character with merges that reach only one and a half bytes pays pieces per character against English's , so eight times the pieces. Nothing about the meaning has changed. The tokenizer has converted the corpus composition into a bill.
Where it stops being simple
Two qualifications matter. If the corpus were balanced across scripts, the merge side of the ratio would move toward those scripts and the cost gap would narrow, though the encoding's one, two, three and four bytes per character would remain. And because Unicode allows several byte sequences for the same visible text, a normalisation step must run before tokenisation. Rewrite the text to one canonical byte sequence and the duplicate forms collapse; skip it and the same sentence from a word processor and from a keyboard can behave differently, for reasons invisible on screen. Normalisation is frozen with the vocabulary: change it after training and the model receives inputs it has never seen.
Worth keeping
- Every string is a sequence of bytes, so a vocabulary of all 256 byte values can represent any input.
- The cost of a script is set by its bytes per character and by how many merges the corpus taught.
- Two visually identical strings can be different byte sequences, so normalisation must happen before tokenisation.
Where this goes next
This question is one step of Cutting Text Into Pieces, a course that works through its subject from the beginning. The page that teaches this part of it is Bytes Underneath Everything, and it reads the whole way through whether or not you have an account.
What an account changes is the practice: problems standing under every idea, marked with a reason rather than a tick, in a copy of the course that is yours.
Read that pageThe whole courseOther questions people ask about this
- Why does the same request cost more tokens in some languages than in others?A token is learned from a corpus, so a language that was rare in the training text is split into many more pieces for the same meaning, and the price multiplier follows.
- Why is 'hello' a different token from ' hello' with a space in front?A space is a byte that gets merged onto the front of the next word, so the tokenizer stores ' hello' and 'hello' as separate symbols with no shared structure.
- Why can't a language model just treat each whole word as one token?Because a fixed list of words cannot contain a word nobody has written yet, and enlarging the list never catches up, while the model still has to put a number to every input it is given.
- Why can't you just add new tokens to a tokenizer after training?Because each token id is tied to weights the model has already trained, so changing the vocabulary makes old numbers name different pieces while the model keeps the old meanings.
- Why does the model get stuck repeating the same sentence over and over?The model is not stuck in a memory loop: once it has written a phrase, that phrase is in its context and makes writing it again more likely, so a decoder that always picks the top word cannot leave.
- Why does diffusion add Gaussian noise instead of any random corruption?Diffusion adds Gaussian noise because Gaussian errors combine exactly: each step keeps the spread fixed and gives a closed-form noisy picture at any later step, which makes the reverse step learnable.