Answers · Putting a model into service

Why does a long conversation with a model use so much more memory than a short one?

In shortThe model keeps a growing summary for every word inside every layer, so a long chat uses memory in proportion to its length, not to the size of its text.

A long conversation uses more memory because the model does not hold the chat as text. It holds, for every token and every layer, a pair of vectors that let later tokens attend to that token without recomputing it. That store grows linearly with the number of tokens, so doubling the conversation roughly doubles this memory, and at long lengths it reaches gigabytes for one conversation.

S = 2 \, L \, d \, g \, f \, \ell

The size of one position

Call the number of layers $L$ , the model width $d$ , the bytes per stored number $f$ , and the factor left after sharing $g$ . The two at the front are the two vectors kept per position: one used to decide attention, one used to carry content. With $L=80$ , $d=8000$ , $f=2$ and no sharing, so $g=1$ , one position costs $2 \times 80 \times 8000 \times 2 = 2{,}560{,}000$ bytes, or 2.56 MB. A conversation of 4,096 positions costs $2{,}560{,}000 \times 4096 = 10{,}485{,}760{,}000$ bytes, about 10.5 GB. The weights of the same model at two bytes a number are 140 GB, so $140 / 10.5 = 13.3$ : thirteen such conversations hold as much as the whole model.

FIG 1How the store grows with the conversation

every head keeps its own pairpairs shared eight heads to a group

Two straight lines, because every extra thousand positions adds the same 2.56 GB without sharing and the same 0.32 GB with eight-way sharing. There is no length at which the line bends.

The plot is two straight lines because every extra thousand positions adds the same 2.56 GB in the unshared case and 0.32 GB in the eight-way shared case. At 32,000 positions those are $2.56 \times 32 = 81.9$ GB and $0.32 \times 32 = 10.24$ GB for one conversation. The absence of a bend is the point: there is no context length at which this store stops growing, only a length at which the machine runs out.

Why it is kept per layer

Each layer looks back separately, so the multiplier $L$ is not an implementation detail. It is one pair of vectors per layer per token. Those vectors depend only on the token and what came before it. They were computed when the token was first processed, and later tokens do not change them. Recomputing instead of keeping would mean the five hundredth token redoing summaries for 499 positions and the five hundred and first redoing 500. For 4,000 tokens the total is $1+2+\cdots+3{,}999 = 7{,}998{,}000$ pair computations instead of 4,000, so the work grows with the square of length. Keeping the pairs turns that quadratic time bill into a linear memory bill.

The natural guess is that the memory is the transcript. A 4,000 word transcript at about five bytes a word is 20,000 bytes, or 20 KB. The store above is 10,485,760,000 bytes, which is $10{,}485{,}760{,}000 / 20{,}000 = 524{,}288$ times larger. Most of that factor is the per layer width: every token is not one word-sized object in memory but $2 \times 80 \times 8000$ numbers, before the bytes-per-number factor. So the surprise is not that the text is long. It is that the text has been expanded into a numerical summary at every layer. Sixty-four conversations at 10.5 GB each need $64 \times 10.5 = 672$ GB, more than a 640 GB machine before any working space.

What changes the number

Sharing summaries between heads is the largest saving. If eight heads share one pair, $g=1/8$ , and 2.56 MB per position becomes $2.56 / 8 = 0.32$ MB; the 4,096 position conversation becomes $0.32 \times 4096 = 1{,}310{,}720{,}000$ bytes, about 1.31 GB. Storing one byte a number instead of two halves it again to 0.16 MB per position and about 0.66 GB. Length is the term the user controls, which is why a product limit on conversation length is often a memory budget rather than a property of the model.

The linear growth is not a law of language. It is a consequence of keeping every position. A system that recomputed the pairs would have fixed memory but quadratic time, which is why production systems do not do it. A system that discards old positions or compresses them into a fixed state does not pay this bill either, but it no longer attends exactly to what was discarded. Within the keep-everything design, the store is proportional to length, and there is no length at which the line bends.

Worth keeping

The memory a long conversation uses is set by tokens and layers, not by the bytes of the transcript.
For eighty layers, width eight thousand and two bytes a number, one position costs 2.56 MB, so 4,096 positions cost about 10.5 GB.
Sharing summaries eight heads to a group divides that store by eight.

Where this goes next

This question is one step of What It Costs to Run a Model, a course that works through its subject from the beginning. The page that teaches this part of it is The Cost of Remembering, and it reads the whole way through whether or not you have an account.

What an account changes is the practice: problems standing under every idea, marked with a reason rather than a tick, in a copy of the course that is yours.

Read that page The whole course