Answers · How models work
Why does the same request cost more tokens in some languages than in others?
In shortA token is learned from a corpus, so a language that was rare in the training text is split into many more pieces for the same meaning, and the price multiplier follows.
A token is not a fixed unit of meaning. It is a piece learned from a corpus, and the same translated sentence takes more pieces in a language the corpus barely contained, so the bill rises by the same factor. For a document of about a thousand English words, the same content took 1,300 pieces in English and 12,350 in Shan: 12,350 / 1,300 = 9.5, so the Shan request costs 9.5 times as much at any fixed price per piece. It also takes 9.5 times as many generation steps and fills 9.5 times as much of a context window, because speed and context are counted in the same pieces.
| Language | Pieces | Multiple against English | Cost in cents |
|---|---|---|---|
| English | 1,300 | 1.00 | 0.65 |
| Spanish | 1,950 | 1.50 | 0.98 |
| Russian | 3,640 | 2.80 | 1.82 |
| Hindi | 6,370 | 4.90 | 3.19 |
| Burmese | 9,880 | 7.60 | 4.94 |
| Shan | 12,350 | 9.50 | 6.18 |
What is being counted
The measurement is not words and not bytes. Words differ in what they carry, and some languages build compounds that other languages would write as several words. The workable comparison is parallel text: the same content translated by people, counted per sentence, then divided by the English count. That gives a ratio for each language, and the ratio is the multiplier on price, latency and context because all three are counted in pieces. A ratio of 1.5 means a sentence that takes 100 pieces in English takes 150 in that language; a ratio of 9.5 means 950.
The arithmetic
Take Shan. The document takes 12,350 pieces. At five dollars per million pieces, the cost is dollars, or 6.175 cents. English takes 1,300 pieces, so dollars, or 0.65 cents. Divide the two bills: . The ratio from the piece counts and the ratio from the bills are the same number, which is what it means to say the tokenizer is the price. The provider is not charging a language surcharge. It is charging for pieces, and the Shan request really is that many pieces long.
The natural guess is that the script is to blame: Burmese and Shan are written in non-Latin characters, so each character costs more bytes, and surely that is the expense. But the byte cost of a script is bounded and small compared with what is observed. A character that takes a few bytes can force a factor of a few at the very worst, and only when the tokenizer has fallen back to individual bytes for everything. The Shan ratio here is 9.5 and the Burmese ratio is 7.6. The script cannot supply the rest. What supplies it is the merge table.
Where the gap comes from
A tokenizer is built by counting pairs of symbols in a corpus and merging the most frequent pair, over and over. Each merge is a piece of vocabulary, and the counting decides who gets it. A language that is a small share of the corpus wins a small share of the merges, so its common words never become single pieces. Its text stays close to the byte floor: short pieces, often single bytes or fragments of characters, where English has whole words and common phrases. This is why the same meaning is long in Shan and short in English, and why the length is not a property of Shan.
- the content that fits, in English-equivalent pieces
- the advertised window, counted in pieces
- the piece ratio for this language against English
This stops being a fact about languages when the vocabulary changes. Fit the merges on a balanced corpus, or reweight the counts so a small language is not swamped, and the ratios flatten: the same Shan text gets longer pieces and the multiple falls towards the European range. The languages have not changed. The counting has. A different tokenizer will give different numbers, so the exact multiple is not eternal, but the mechanism is: any vocabulary that spends its merges where the text is plentiful will make the sparse languages expensive. The repair is to spend some merges where the text is scarce, and to measure the ratio across the languages you actually serve before the vocabulary is fixed.
Worth keeping
- Price, latency and context are all counted in the same unit, so a language that needs 9.5 times as many pieces costs 9.5 times as much at a fixed price per piece.
- A script's byte cost can explain a small factor at most, not the factor of seven or nine seen in the piece counts.
- Fitting merges on a balanced or reweighted corpus flattens the ratios without changing the languages.
Where this goes next
This question is one step of Cutting Text Into Pieces, a course that works through its subject from the beginning. The page that teaches this part of it is What It Costs in Other Languages, and it reads the whole way through whether or not you have an account.
What an account changes is the practice: problems standing under every idea, marked with a reason rather than a tick, in a copy of the course that is yours.
Read that pageThe whole courseOther questions people ask about this
- Why do tokenizers work on bytes instead of on characters?Tokenizers use bytes because 256 byte values can represent any text, while Unicode has too many characters to include safely.
- Why is 'hello' a different token from ' hello' with a space in front?A space is a byte that gets merged onto the front of the next word, so the tokenizer stores ' hello' and 'hello' as separate symbols with no shared structure.
- Why can't a language model just treat each whole word as one token?Because a fixed list of words cannot contain a word nobody has written yet, and enlarging the list never catches up, while the model still has to put a number to every input it is given.
- Why can't you just add new tokens to a tokenizer after training?Because each token id is tied to weights the model has already trained, so changing the vocabulary makes old numbers name different pieces while the model keeps the old meanings.
- Why does the model get stuck repeating the same sentence over and over?The model is not stuck in a memory loop: once it has written a phrase, that phrase is in its context and makes writing it again more likely, so a decoder that always picks the top word cannot leave.
- Why does diffusion add Gaussian noise instead of any random corruption?Diffusion adds Gaussian noise because Gaussian errors combine exactly: each step keeps the spread fixed and gives a closed-form noisy picture at any later step, which makes the reverse step learnable.