What a Million Words Actually Costs
Last timeTwo Phases, Two Bottlenecks
Everything in this course assembles into a single number, a price per million words, and that number ranks the available changes honestly.
Everything in this course has been a term in one calculation. This lesson runs
it, and then uses it to put the available changes in order.
- what the machine costs for an hour, which for eight chips is in the region of twenty-four dollars
- the seconds one pass takes, which is the bytes moved divided by the memory bandwidth
- how many requests are grouped, since one pass produces one word for each of them
- seconds in an hour, converting the hourly price into a price per second
- scaling the answer up to a million words, which is the unit prices are usually quoted in
- the cost of producing a million words, before anything is marked up
The four steps
Take the machine this course has been using throughout: a hundred and forty
gigabytes of weights, three thousand gigabytes a second of memory bandwidth,
sixty-four conversations grouped, each holding one and a third gigabytes. One
pass moves two hundred and twenty-six gigabytes and takes seventy-five
milliseconds, producing sixty-four words. That is eight hundred and fifty words a
second, three million words an hour, and at twenty-four dollars an hour it comes
to about eight dollars a million words.
It is worth pausing on how little that calculation needed. No profiler, no
benchmark run, no access to the machine. Four numbers about the model, two about
the hardware and one about the group, and the result is within a small factor of
what a well-run service will actually achieve. That is unusual in computing, and
it is a consequence of the whole system being limited by one resource whose rate
is published on a specification sheet.
| step | gigabytes moved per pass | milliseconds per pass | words per second | dollars per million words | what happened |
|---|---|---|---|---|---|
| 1 | 226 | 75 | 853 | 7.8 | The starting point: two bytes a number, summaries shared eight ways, sixty-four grouped. |
| 2 | 113 | 37.7 | 1698 | 3.9 | One byte a number instead of two. Both the weights and the stores halve, so the whole pass halves. |
| 3 | 483 | 161 | 1590 | 4.2 | Back to two bytes but with two hundred and fifty-six grouped. Each pass is much slower and produces four times as many words. |
| 4 | 242 | 80.7 | 3172 | 2.1 | Both changes at once. They multiply rather than overlapping, because one shrinks the bytes and the other spreads them. |
| throughput against the s | dollars per million word | |
|---|---|---|
| one byte a number instea | 2.00 | 3.90 |
| group of two hundred and | 1.86 | 4.20 |
| summaries shared sixty-f | 1.49 | 5.20 |
| half the conversation le | 1.23 | 6.30 |
| the starting point | 1.00 | 7.80 |
What each one is paid for with
The ranking above is only half an answer, because the column it leaves out is
what you give up.
The lesson stops here
2 more paragraphs to go
You have read the opening. The rest of the argument, the problems that check whether it landed, and the lines worth keeping at the end all come with a plan.
The first lesson of every course in the library reads the whole way through, free, so you can see exactly what the rest of them are.
See the planThe contentsThis is the reading half
Starting the course gives you your own copy of it. Every idea on every page has problems standing under it, marked with a reason rather than a tick, and any sentence you do not believe can be opened and argued with. None of that can happen on a page nobody owns.
The contents