Answers · Putting a model into service

Why does quantization-aware training use fake quantization instead of just quantizing at the end?

In shortFake quantisation lets training see the rounded weights in the forward pass while still updating the exact weights, so the model learns weights that survive rounding instead of being rounded only after training.

Quantisation-aware training uses fake quantisation because the aim is not to round a finished model but to move the weights before they are rounded. In the forward pass each weight is replaced by its rounded value, so the loss is the loss of the model that will actually be served. In the backward pass the gradient is passed through the rounding unchanged, so the full-precision weights can still move. Training therefore finds weights whose nearest representable value is nearly as good as the exact one, which a model trained without rounding has no reason to have done.

\hat w = s\,\operatorname{clip}\!\left(\operatorname{round}\!\left(\frac{w}{s}\right),-q_{\max},q_{\max}\right), \qquad \frac{\partial L}{\partial w} \approx \frac{\partial L}{\partial \hat w}.

The derivative is the obstacle, not the verdict

The natural guess is that rounding has zero derivative, so training cannot help. The first half is right. The derivative of round is zero almost everywhere, and if backpropagation used the true derivative the gradient arriving at every weight would be multiplied by zero and nothing would learn. But that is an argument against using the true derivative, not against training. Fake quantisation defines the backward pass to be the identity: the gradient with respect to the rounded weight is handed back to the full-precision weight as though the rounding were not there. It is an approximation, not the true derivative, and it is the approximation that makes the method work. The forward pass is not softened either. It uses exactly the rounded value, the same arithmetic that will run at deployment.

Take a weight $w=0.1234$ and a step size $s=0.05$ . The forward pass computes $w/s=2.468$ , rounds that to $2$ , and uses $\hat w=2\times0.05=0.1$ . The loss therefore sees an error of $0.1234-0.1=0.0234$ for that weight. The backward pass does not differentiate the rounding; it sends the gradient of the loss with respect to $0.1$ straight back to $0.1234$ . The update is applied to the full-precision weight, so the next forward pass can land on a different grid point. A finished model never gets that chance: its weight is fixed at $0.1234$ and the nearest grid point is simply accepted.

That is the whole difference. Rounding after training takes the weights as fixed and asks only which grid point is nearest. Fake quantisation asks, during training, which full-precision weight would make its nearest grid point good. The final bit width is the same in both cases. The weights are not. A useful way to put it is that the cheap method minimises a loss that never contains a rounding error, while fake quantisation minimises a loss that already contains the error the deployed model will have. The derivative is approximate; the objective is the right one.

What the extra run buys, and when it does not

The cost is a training run, not an hour of calibration. The return is largest where rounding hurts most. At eight bits the rounded and rounding-aware models are close enough that the training run buys a difference nobody can measure, so rounding the finished model is the right default. At four bits the cheap method is still usable for many models, but the gap is real and a short repair run on representative data often recovers most of it without a full run. Below four bits the cheap method stops working sharply, and anything in that territory has fake quantisation somewhere in its training.

Read the gain horizontally rather than vertically. Training with rounding lets three-bit storage behave roughly like four-bit storage, which is the two bits of headroom. That is worth a run only if you need those bits. If you are being quoted three-bit or two-bit results, the first question is whether the model was merely rounded after training or trained with the rounding in the loop, because the same bit width means quite different quality depending on the answer.

Where it stops being true

Two limits are worth stating. The straight-through estimator is not a true gradient, so fake quantisation is an approximation with no guarantee; bad step sizes or extremely low bit widths can make training unstable. And fake quantisation does not remove the need for representative data. If activations are quantised too, their ranges still have to be chosen, and a calibration set that does not look like production traffic will set ranges that clip the values that matter. Training the weights to tolerate rounding does not fix a range chosen on the wrong inputs.

Worth keeping

Rounding a finished model is cheap and adequate at eight bits, but it cannot move weights that were chosen without rounding in mind.
Fake quantisation uses rounded weights in the forward pass and an identity gradient in the backward pass, so training optimises the model that will ship.
The gain from training with rounding is about two bits, and it is not worth a training run when eight-bit rounding already costs almost nothing.

Where this goes next

This question is one step of Making a Model Smaller, a course that works through its subject from the beginning. The page that teaches this part of it is Rounding After, or Training With, and it reads the whole way through whether or not you have an account.

What an account changes is the practice: problems standing under every idea, marked with a reason rather than a tick, in a copy of the course that is yours.

Read that page The whole course