Answers · Putting a model into service
Why doesn't pruning half the weights make the model twice as fast?
In shortBecause the zeros stay in the matrix: they are still stored and still multiplied, so the model is the same size and the same speed as before.
Pruning half the weights changes none of the numbers the machine actually handles. The matrix keeps its shape, so it is stored at full size and multiplied at full speed; what gets removed only makes the model smaller when it is removed in whole rows, columns, heads or layers.
Take a small block and set the smallest weights to zero. Eighteen of the thirty-six entries below are gone, and every one of them sits exactly where it sat before.
| c1 | c2 | c3 | c4 | c5 | c6 | |
|---|---|---|---|---|---|---|
| row 1 | 0.41 | 0.00 | 0.00 | 0.63 | 0.00 | 0.22 |
| row 2 | 0.00 | 0.37 | 0.58 | 0.00 | 0.19 | 0.00 |
| row 3 | 0.26 | 0.00 | 0.00 | 0.00 | 0.44 | 0.31 |
| row 4 | 0.00 | 0.51 | 0.33 | 0.28 | 0.00 | 0.00 |
| row 5 | 0.18 | 0.00 | 0.47 | 0.00 | 0.00 | 0.39 |
| row 6 | 0.00 | 0.29 | 0.00 | 0.36 | 0.55 | 0.00 |
Count the arithmetic instead of trusting the percentage. The first row of the pruned block produces its first output like this:
Six multiply-accumulates, three of them against zero, exactly as before pruning. The other five rows contribute five more apiece, so the block performs 36 multiply-accumulates where it performed 36 last week. Memory follows the same rule. A 4096 by 4096 weight matrix in fp32 holds numbers at 4 bytes each, which is 67,108,864 bytes, or 64 MiB. Zeroing half of them, 8,388,608 entries, leaves the allocation at 64 MiB, because the tensor was allocated as a rectangle and the zeros were written into slots that had already been paid for.
The natural guess is that a multiply by zero is free, or that the hardware notices the zero and skips that slot. It does not do either. A dense kernel loads a fixed number of values into fixed-width lanes and issues one multiply-accumulate per lane, whatever values arrive. Testing each weight before multiplying it would cost a branch, and a branch per weight destroys the pipelining that makes the kernel fast in the first place, so no production kernel does it. Multiplication by zero is one instruction, the same instruction as multiplication by anything else.
Measured on the same model, the two families of removal come apart exactly as the arithmetic predicts.
| Method | Memory freed, per cent | Speed gained, per cent | Quality cost, points | Needs special hardware |
|---|---|---|---|---|
| Scattered, half the weights, dense storage | 0 | 0 | 1.4 | 0 |
| Scattered, half the weights, sparse format | 34 | -12 | 1.4 | 0 |
| Scattered, fixed two-in-four pattern | 0 | 38 | 2.1 | 1 |
| Whole rows and columns, a fifth removed | 20 | 19 | 3.6 | 0 |
| Whole attention heads, a quarter removed | 9 | 11 | 2.8 | 0 |
| Whole layers, a sixth removed | 17 | 17 | 5.9 | 0 |
Read the first row and stop: half the weights gone, nothing freed, nothing faster, a point and a half of quality spent for no return in either currency. The second row is the usual escape, storing only the survivors together with their positions. It does free memory, 34 per cent of it, but the surviving values now sit at irregular addresses, so the kernel spends its time gathering them, and that row runs 12 per cent slower than the dense matrix it replaced. A saving in bytes that costs you speed is not the trade you thought you were making.
Why only the structured rows move
When a whole row or column is deleted, the matrix is genuinely one row shorter, its output is one value narrower, and whatever consumes that output shrinks to match. That is why memory and speed finally agree in the table: 20 per cent of the rows and columns removed frees 20 per cent of the memory and saves 19 per cent of the time, because there is 20 per cent less matrix. The price is that you cannot pick only the useless weights. A whole row goes, and it takes the useful weights in that row with it.
That price is steep. At a fifth removed, the scattered cost works out at points, while the structured cost is points.
So the same weight saving costs about 2.7 times as much quality, and the gap widens with the dose: at 50 per cent removed the structured curve is at 38 points of damage against 3.1 for the scattered one. That is why structured removal is quoted in doses of ten to twenty per cent while scattered removal is quoted at fifty. The percentage is chosen by the family, not by the model.
Worth keeping
- Setting a weight to zero does not change the shape of a matrix, so the zeros are stored and multiplied like any other number.
- Removing whole rows, columns, heads or layers does shorten the arithmetic and free memory, and costs more quality per weight removed.
- Removing the same amount in several stages with a short repair run between them beats removing it all at once by roughly a factor of seven in quality lost.
Where this goes next
This question is one step of Making a Model Smaller, a course that works through its subject from the beginning. The page that teaches this part of it is Taking Weights Out Entirely, and it reads the whole way through whether or not you have an account.
What an account changes is the practice: problems standing under every idea, marked with a reason rather than a tick, in a copy of the course that is yours.
Read that pageThe whole courseOther questions people ask about this
- Why does the memory used by a model keep growing as the conversation gets longer?The model's weights do not grow. It keeps a cache of keys and values for every earlier token, so memory rises with conversation length and with how many conversations run at once.
- Why does one outlier weight wreck 4-bit quantization for the whole layer?A single huge value sets the rounding step for the whole layer, so ordinary weights fall below one step and round to zero.
- What do the scale and zero-point actually do in 4-bit quantization?The scale sets the spacing between the sixteen levels, and the zero-point slides the whole grid so it lines up with the weights you actually have.
- Why does quantization-aware training use fake quantization instead of just quantizing at the end?Fake quantisation lets training see the rounded weights in the forward pass while still updating the exact weights, so the model learns weights that survive rounding instead of being rounded only after training.
- Why does the model return JSON wrapped in extra text when I asked for JSON?A request for JSON does not force the model to return only JSON, so you need to validate the reply and repair it once rather than assume it will parse.
- Why do the numbers look good while users keep complaining?Because the score measures a fixed set of cases with a fixed rule, and the users are living through behaviour that the set and the rule never covered.