Answers · Putting a model into service

At what point does the conversation history cost more than the model itself?

In shortWith sixty-four conversations grouped together, reading their history overtakes reading the model at roughly six and a half thousand words per conversation.

For a hundred and forty gigabytes of weights, sixty-four grouped conversations and a third of a megabyte kept per position, the crossover is about 6,563 positions per conversation. Past that length, reading the kept history of the group costs more bytes for each new word than reading the model itself.

That crossover is the speed cost, not the capacity cost. The same kept positions also occupy memory, so fewer conversations fit and fewer can be grouped. Capacity failure looks like queues or refusals while the machine is underworked; speed failure looks like every reply arriving slowly with nothing refused. The 6,563-position crossover is the speed one.

The arithmetic

To produce one new word, the machine reads the weights once for the whole group and then reads every kept position of every conversation in that group. If $M$ is the bytes moved per new word, $W$ the weight bytes, $B$ the number grouped, $s$ the bytes kept per position for one conversation and $\ell$ the current length of each conversation, then

M = W + B s \ell

The weight term is unchanged by length or by group size. The history term is not: it is multiplied by both the number of conversations and how far each has already run. With $W=140$ GB $=140{,}000$ MB, $s=\tfrac13$ MB and $B=64$ ,

B s = 64 \times \tfrac13 \text{ MB} = 21.333 \text{ MB} = 0.021333 \text{ GB per position}

\ell^{*} = \frac{W}{B s} = \frac{140{,}000 \text{ MB}}{21.333 \text{ MB}} = 6{,}562.5

So the two terms are equal at 6,562.5 positions, or 6,563 in whole positions. Below that the weights dominate; above it the group's stores dominate.

At 1,000 positions the group's history costs $21.333 \times 1{,}000 = 21{,}333$ MB, or 21.3 GB, about 15 per cent of the 140 GB of weights. At 10,000 positions it costs 213.3 GB, which is already larger than the weights. The growth is linear in length, not flat, so the same machine at 10,000 positions is spending more bytes on the conversation for one new word than it spends on the model.

Why the flat guess fails

The natural guess is that an extra word adds a fixed amount of delay, so a long conversation is merely slower by a constant penalty per word. The penalty per new word at length $\ell$ is $B s \ell$ , which is proportional to how far the conversation has already gone. At the hundredth position one conversation adds $100 \times \tfrac13$ MB $=33.3$ MB to the read; at the ten-thousandth it adds $10{,}000 \times \tfrac13$ MB $=3{,}333$ MB, or 3.33 GB. Grouping makes the same point stronger, because every conversation in the group brings its own growing store to every step.

FIG 1Bytes read per new word against conversation length

the weightsthe stores of sixty-four grouped conversations

The weights are a flat 140 GB per new word for the group. The history term is 64 conversations times a third of a megabyte per position, so it is 21.333 MB per position and crosses 140 GB at about 6,563 positions.

The formula shows directly which settings change the answer. For a single conversation, $B=1$ , so $\ell^{*}=140{,}000/(1/3)=420{,}000$ positions. Grouping sixty-four conversations divides that by 64 and gives 6,562.5. Halving $s$ doubles the crossover; sharing summaries eight ways divides the store term by eight and multiplies the crossover by eight. Raising the group size to improve throughput lowers it in direct proportion, which is the tension: the same change that makes the weights cheaper to share makes long conversations arrive at their dominant cost sooner.

Where the shape changes

Producing words one at a time is linear in the stored length, but reading a prompt is quadratic. All of its positions arrive together and each must be compared with every earlier one, so the comparisons grow with $\ell^2$ while the rest of the work grows with $\ell$ . The two are equal at roughly six times the model width. For a model of width 8,000, that is about 48,000 positions. At 50,000 positions the comparison share is $50/(49.15+50)=0.504$ , just over half the arithmetic; at 100,000 it is $100/(49.15+100)=0.671$ .

Tiling the comparisons keeps the table in fast memory and removes the quadratic movement while producing exactly the same answer. Sharing summaries between heads reduces $s$ and moves both crossovers out. Discarding older positions bounds the store at any length, but it changes what the model answers, and a system can then forget text inside a context limit it still advertises. The first repair is free in the result; the second was paid for at training time; the third is a product decision. A visible length limit is honest; silent forgetting inside a long advertised context limit is not.

Worth keeping

With 140 GB of weights, 64 grouped conversations and a third of a megabyte kept per position, the history of the group costs as much to read as the weights at 6,563 positions per conversation.
Grouping more conversations improves throughput but lowers this crossover in direct proportion: 64 conversations put it at one sixty-fourth of the single-conversation value of 420,000 positions.
Reading a prompt is quadratic, so the pairwise comparisons become half the arithmetic at about 50,000 positions for a model of width 8,000.

Where this goes next

This question is one step of What It Costs to Run a Model, a course that works through its subject from the beginning. The page that teaches this part of it is When the Context Gets Long, and it reads the whole way through whether or not you have an account.

What an account changes is the practice: problems standing under every idea, marked with a reason rather than a tick, in a copy of the course that is yours.

Read that page The whole course