Answers · Putting a model into service

Why is writing a long answer slower than reading a prompt of the same length?

In shortReading a prompt happens in one parallel pass, while writing a reply happens one word at a time, so the reply rereads the model for every word.

Reading a prompt of two thousand words takes one pass through the model. Writing a reply of two thousand words takes two thousand passes, because the reply is produced one word at a time and each pass reads the weights again.

I_{r}=B\ell,\qquad I_{w}=B

Here $B$ is the number of requests grouped into a pass and $\ell$ is the number of prompt positions taken in. With one request, $B=1$ . For two thousand words, $\ell=2000$ . Reading gives $I_{r}=1\times 2000=2000$ operations per byte of weights moved. Writing gives $I_{w}=1$ operation per byte. The reading phase does two thousand times as much arithmetic per byte moved, because it moves the weights once and uses them for every prompt position.

Each new reply word depends on all the words before it, so the next pass cannot start until the previous word exists. That serial dependency is why the reply is a sequence of passes rather than one pass with many positions. The prompt has no equivalent dependency for the purpose of taking it in: every prompt position can be multiplied by the same weight matrix in the same pass.

FIG 1Reading and writing two thousand words

step	passes through the weights	seconds of arithmetic	seconds of movement	what happened
1	1	0.09	0.05	Taking in a two hundred word prompt. One pass, and arithmetic already exceeds movement.
2	1	0.94	0.05	Taking in a two thousand word prompt. Ten times the arithmetic and the same movement, because the weights were read once either way.
3	200	0.09	9.3	Producing a two hundred word reply. Identical arithmetic to the first row, and nearly two hundred times its movement.
4	2000	0.94	93.4	Producing a two thousand word reply. The same arithmetic as the second row and a hundredfold the time.

4 steps

Rows two and four do the same arithmetic. The reply row is slow because it moves the weights two thousand times.

The same arithmetic, different movement

Rows two and four do exactly the same arithmetic, $0.94$ seconds each. The prompt row takes $0.05$ seconds of movement; the reply row takes $93.4$ seconds. The difference is $93.4-0.05=93.35$ seconds, and all of it comes from reading the weights two thousand times instead of once. The arithmetic did not grow, the movement did.

What the guess misses

The natural guess is that a word costs the same to read and to write, so two thousand words in and two thousand words out should take similar time. That would be true only if both phases did the same number of passes. They do not. The prompt is one pass with $\ell$ positions inside it; the reply is $\ell$ passes with one position inside each. Multiplying a single per-word cost by length in both directions is the mistake.

FIG 2

what differs	taking in a prompt	producing words
what limits it	arithmetic	moving weights and stores
passes per word	one pass for the whole prompt	one pass per word
what the user feels	the wait before the first word	the pace words arrive
best group size	small, already saturated	as large as memory allows
better hardware	more arithmetic per second	more memory bandwidth and capacity

The user feels the two phases differently. The prompt length sets the wait before the first word appears. The reply length sets the pace at which later words arrive. A long prompt makes a reader wait once; it does not make each reply word slower. Averaging the two into one latency number hides the part the reader actually notices.

This stops being a fixed ratio when replies are grouped. Writing with $B$ requests in a pass has $I_{w}=B$ operations per byte. Take $B=300$ : writing now has $300$ operations per byte, against $1$ for a single request, so it can cross the same balance point a long prompt crosses on its own. More requests per pass can then raise throughput without rereading the weights for each request. The cost moves to memory capacity and to the wait that each request sees.

The phases also interfere when run on one machine. A pass is indivisible: if it is spent taking in a long prompt, it produces no word for any reply already in flight. That is why a long prompt can make every other reader's stream pause, even though the prompt is cheap per word. If words arrive twenty a second, a half second pause is $20\times 0.5=10$ missing words, which is plainly visible. The repairs are to cut the prompt into fixed pieces so no pass is consumed by it, or to put the two phases on separate machines.

Worth keeping

Reading a prompt pushes every position through the weights in one pass; writing a reply pushes one position per pass.
With one request, a two thousand word prompt is about two thousand operations per byte to read and one operation per byte to write.
Grouping many replies can make writing arithmetic-bound, but it does not make the prompt and reply phases cost the same per word.

Where this goes next

This question is one step of What It Costs to Run a Model, a course that works through its subject from the beginning. The page that teaches this part of it is Two Phases, Two Bottlenecks, and it reads the whole way through whether or not you have an account.

What an account changes is the practice: problems standing under every idea, marked with a reason rather than a tick, in a copy of the course that is yours.

Read that page The whole course

Why is writing a long answer slower than reading a prompt of the same length?

The same arithmetic, different movement

What the guess misses

Worth keeping

Where this goes next

Read alongside