Why does one outlier weight wreck 4-bit quantization for the whole layer?

Take a layer whose weights are mostly around 0.9, with one weight at 112. The largest magnitude is 112, so $s=112/7=16$. For $w=0.9$, $w/s=0.05625$, which rounds to 0, so the output is 0 and the error is 0.9. For $w=3.4$, $w/s=0.2125$, which also rounds to 0, so the error is 3.4. Now hold the outlier aside. The largest remaining magnitude is 3.4, so $s=3.4/7=0.485714$. The same 0.9 gives $0.9/0.485714=1.852$, which rounds to 2, so the output is $2\times0.485714=0.971429$ and the error is $0.971429-0.9=0.071429$. The same 3.4 gives exactly 7 steps and is represented exactly. The step has fallen by a factor of $16/0.485714=32.9$, and the typical weight's error has fallen from 0.9 to 0.071429, a factor of $0.9/0.071429=12.6$.

The whole layer suffers because a per-tensor scale is shared. The outlier does not need to be large in error terms; it only needs to be large enough to set the step for every other weight. With four bits there are only $2^4=16$ levels across the full range, so once the range is stretched to cover 112 the bins between levels are wider than most of the weights that need representing.

The same arithmetic applies to any group of numbers that share a scale, but in large transformers the extreme values that actually wreck a layer are usually in the activations, not in the stored weights. A small number of channels carry values one or two orders of magnitude above the rest for every input. If you choose the activation scale from the maximum, that one channel stretches the range and the ordinary channels lose resolution.

That matters because a weight that looks ordinary can be important through what it multiplies. Suppose a weight of 0.9 is multiplied by an activation of 112. Its product is $0.9\times112=100.8$; the same weight multiplied by a typical activation of 0.9 gives $0.9\times0.9=0.81$. An error in that weight is multiplied by 112, not by 0.9, so a sensitivity sweep that looks only at stored weights will rank it as harmless.

The failure is not a property of 4-bit arithmetic in the abstract; it is a property of sharing one scale. If you quantise per channel, an outlier damages its own channel and leaves the others alone. If you clip the top of the range, you trade a little saturation on the outlier for much finer steps everywhere else, which can be the better trade when the outlier is rare. If you hold the extreme channel aside and keep it at higher precision, the remaining range collapses and the ordinary weights quantise cleanly. The cost of that exception is the memory you give back, and it is usually small because the exception is a small fraction of the weights.

The practical consequence is that a uniform bit width is always wrong somewhere. The damage is not spread evenly across a model: rounding one part and leaving everything else exact shows some parts cost an order of magnitude more than others. The right question is not whether to use four bits, but which parts to keep precise. Measure one part at a time, rank by cost per per cent of weights, then measure the whole plan together, because individual costs do not simply add.

A single exception can be cheap and decisive. Keeping a small sensitive fraction at sixteen bits and the rest at four still gives most of the memory saving, while the first few per cent of exceptions recover most of the quality.