The Same Sentence, Nine Times the Bill
Last timeNumbers, Code and Other Awkward Text
Price, speed and context are all counted in pieces, and the same meaning takes several times more pieces in most languages than in English. Here is how much, and why.
Measuring it properly
The question is how many pieces a language needs for a given amount of
meaning. That is harder to measure than it sounds, because meaning has no
units.
Comparing arbitrary documents will not do: a newspaper in one language and a
novel in another differ for reasons that have nothing to do with the tokenizer.
Comparing per word will not do either, since a word means different things in
different languages and some, like German and Finnish, build compounds that
would be several words elsewhere.
The method that works is parallel text. Take a collection translated into many
languages by people, so the content is held as fixed as translation allows, and
count the pieces each version takes. Then divide by the English count. Per
sentence rather than per word, because a sentence is the closest thing to a
constant unit of content across languages.
| pieces for the document | cents at five dollars a | per cent of a context wi | |
|---|---|---|---|
| English | 1300.00 | 0.65 | 100.00 |
| Spanish | 1950.00 | 0.98 | 67.00 |
| Russian | 3640.00 | 1.82 | 36.00 |
| Hindi | 6370.00 | 3.19 | 20.00 |
| Burmese | 9880.00 | 4.94 | 13.00 |
| Shan | 12350.00 | 6.18 | 11.00 |
The bill
The second column of that table is the whole practical point. Providers bill
per piece. There is no adjustment for language, and there is no reason there
would be: the provider really does do several times more work for the Burmese
request, because the sequence really is several times longer.
So the ratio is a price multiplier, exactly. A speaker of a language at four
times the English rate pays four times as much for the same question. They also
wait longer, because generation time is per piece too, and the answer comes back
in the same expensive units as the question.
This compounds in an unpleasant direction. The languages with the highest
ratios are, with few exceptions, the languages of the poorest countries, because
both facts come from the same cause: how much text in that language was on the
public internet when the corpus was collected. The people paying nine times as
much are the people least able to.
The shrinking context
A context window is a number of pieces, so the same arithmetic applies and
nobody mentions it.
The lesson stops here
3 more paragraphs to go
You have read the opening. The rest of the argument, the problems that check whether it landed, and the lines worth keeping at the end all come with a plan.
The first lesson of every course in the library reads the whole way through, free, so you can see exactly what the rest of them are.
See the planThe contentsThis is the reading half
Starting the course gives you your own copy of it. Every idea on every page has problems standing under it, marked with a reason rather than a tick, and any sentence you do not believe can be opened and argued with. None of that can happen on a page nobody owns.
The contents