Answers · Putting a model into service
Why does the memory used by a model keep growing as the conversation gets longer?
In shortThe model's weights do not grow. It keeps a cache of keys and values for every earlier token, so memory rises with conversation length and with how many conversations run at once.
Because the model keeps a cache of the keys and values for every token already in the conversation, and that cache belongs to the request rather than to the weights. The weights stay fixed; the cache grows by a fixed amount for each new token and is paid again for each conversation served at the same time.
- total per-request state, in bytes
- number of layers in the model
- size of the state each layer keeps per token
- tokens in the conversation so far
- requests being served at the same time
- bytes per stored number
The cache is the growth
Put numbers in the formula. A model with forty layers that keeps eight kilobytes of state per token per layer holds 40 times 8 KB = 320 KB for each token, which is about a third of a megabyte. A four thousand token conversation therefore holds 4000 times 0.33 MB = 1320 MB, or 1.32 GB, for one user. Thirty such users at once hold 30 times 1.32 GB = 39.6 GB, which is larger than many of the models people are trying to shrink. The factor of two in the formula is not decoration: each layer keeps a key vector and a value vector for every token, and both are needed to attend to that token later. At thirty-two thousand tokens and thirty-two simultaneous requests the same arithmetic gives 32 times 10.56 GB = 337.92 GB.
| Context length | One request | 32 requests |
|---|---|---|
| 1,000 tokens | 0.33 GB | 10.56 GB |
| 4,000 tokens | 1.32 GB | 42.24 GB |
| 32,000 tokens | 10.56 GB | 337.92 GB |
Those are straight lines because the cache is linear in both context and concurrency. Doubling the conversation length doubles that request's cache. Doubling the number of simultaneous requests doubles the total again. The weights do not appear in this calculation at all, which is why a model can be small enough on disk and still exhaust the machine once a few long conversations are open. That is why the same machine serves many short conversations or a few long ones, but not both at full volume.
The cache exists to avoid recomputing the same arithmetic. Without it, producing token number four thousand would require recomputing the keys and values for the preceding three thousand nine hundred and ninety-nine tokens, and the total work over a conversation would grow with the square of its length. Keeping them trades memory for time. The memory is the price of making long conversations fast, and it is charged per token, per layer, per request, for as long as the conversation is open.
What is not growing
The natural guess is that the model itself is getting bigger, or that a longer context is free once the weights are loaded. Neither is true. The weights are one multiplication, parameters times bytes per parameter: a seven billion parameter model at two bytes per parameter is 7,000,000,000 times 2 = 14,000,000,000 bytes, or 14 GB, at the first token and at the millionth. At four bits per parameter the same weights are 3.5 GB. That number is fixed per machine and per copy of the model; it does not know how long the conversation is. What grows is the cache, and the cache is not the model's knowledge. It is the intermediate state that lets the next token be produced without starting over.
Attacking the weights therefore cannot fix a cache problem. If the machine is running out of room because thirty users each hold a long conversation, halving the weights frees a fixed number of gigabytes once. Halving the cache frees memory per user, every second, and it keeps freeing it as context grows. The two budgets are not interchangeable.
Where the line stops
The growth is linear only while every past token is kept. A sliding window, truncation, or a server that recomputes rather than caches changes the slope, and can cap it. Quantising the cache to eight bits halves the bytes per stored number, so the same conversation costs half as much, but the line still rises. Moving the cache to slower memory or to disk changes where it lives, not how many bytes it needs. And at small context and few users the cache can be so small that the weights are the only budget that matters; the crossover is a traffic level, not a universal rule.
Measure the budgets on the actual serving machine before choosing a technique. Weight memory is parameters times bytes per parameter and is fixed per copy. Cache memory is the formula above, linear in context and concurrency. The budget that is large tells you what to shrink, and the technique with the impressive number attached is often aimed at a different budget. If the cache is the large one, shrinking the weights buys room once and then the next long conversation takes it back.
Worth keeping
- Weight memory is parameters times bytes per parameter and does not change with conversation length.
- Per-request cache memory is linear in context length and in concurrent requests.
- Halving weights does nothing for a system that runs out of room on the key-value cache.
Where this goes next
This question is one step of Making a Model Smaller, a course that works through its subject from the beginning. The page that teaches this part of it is What Is Actually Taking the Room, and it reads the whole way through whether or not you have an account.
What an account changes is the practice: problems standing under every idea, marked with a reason rather than a tick, in a copy of the course that is yours.
Read that pageThe whole courseOther questions people ask about this
- Why does one outlier weight wreck 4-bit quantization for the whole layer?A single huge value sets the rounding step for the whole layer, so ordinary weights fall below one step and round to zero.
- What do the scale and zero-point actually do in 4-bit quantization?The scale sets the spacing between the sixteen levels, and the zero-point slides the whole grid so it lines up with the weights you actually have.
- Why does quantization-aware training use fake quantization instead of just quantizing at the end?Fake quantisation lets training see the rounded weights in the forward pass while still updating the exact weights, so the model learns weights that survive rounding instead of being rounded only after training.
- Why doesn't pruning half the weights make the model twice as fast?Because the zeros stay in the matrix: they are still stored and still multiplied, so the model is the same size and the same speed as before.
- Why does the model return JSON wrapped in extra text when I asked for JSON?A request for JSON does not force the model to return only JSON, so you need to validate the reply and repair it once rather than assume it will parse.
- Why do the numbers look good while users keep complaining?Because the score measures a fixed set of cases with a fixed rule, and the users are living through behaviour that the set and the rule never covered.