Basis Lab

Answers · Putting a model into service

What do the scale and zero-point actually do in 4-bit quantization?

In shortThe scale sets the spacing between the sixteen levels, and the zero-point slides the whole grid so it lines up with the weights you actually have.

The scale sets how far apart the sixteen representable levels sit; the zero-point slides the entire grid sideways until it lies over the weights you actually have. Quantising a weight is therefore two operations before the rounding, not one: divide by the scale, add the zero-point, round, and store that small integer. To compute with it again, subtract the zero-point and multiply by the scale. Both the scale and the zero-point are stored next to the integers, once per group of weights.

q=round⁡(ws)+z,w^=s (q−z)q = \operatorname{round}\left(\frac{w}{s}\right) + z, \qquad \hat{w} = s\,(q - z)

Here ww is the original weight, qq the integer that goes to memory, ss the scale and zz the zero-point. The scale is not a free parameter. It is the span the group has to cover divided by the number of intervals the bits can express, and the zero-point is whatever shift puts the bottom of that span on the lowest code.

s=vmax⁡−vmin⁡2k−1,z=round⁡(−vmin⁡s)s = \frac{v_{\max} - v_{\min}}{2^{k} - 1}, \qquad z = \operatorname{round}\left(\frac{-v_{\min}}{s}\right)

Take a group of 128 weights running from −0.05-0.05 to 0.600.60. It is lopsided: almost nothing lives below zero, which is what a block of activations behind a rectifier looks like. Four bits gives 16 codes. A mapping with no zero-point has to stay centred on zero and still reach 0.600.60, so it spreads codes −7-7 to 77 at a step of 0.60/7=0.08570.60/7 = 0.0857. Its worst rounding error is half of that, 0.04290.0429.

With a zero-point, the step comes from the true span instead: (0.60−(−0.05))/15=0.65/15=0.0433(0.60 - (-0.05))/15 = 0.65/15 = 0.0433, and z=round⁡(0.05/0.0433)=round⁡(1.15)=1z = \operatorname{round}(0.05/0.0433) = \operatorname{round}(1.15) = 1. Check both ends. The weight −0.05-0.05 gives −1.15-1.15, rounds to −1-1, and code −1+1=0-1 + 1 = 0. The weight 0.600.60 gives 13.8513.85, rounds to 1414, and code 14+1=1514 + 1 = 15. The bottom of the data sits on code 0 and the top on code 15, so all sixteen codes are carrying weights and the worst error is half a step, 0.02170.0217.

Read the codes column of that table as the real work the zero-point does. The symmetric grid burns codes −7-7 down to −2-2 on a region where the weights barely reach −0.05-0.05, and one weight, 0.03740.0374, rounds to zero and is deleted rather than approximated. The zero-point does not make the grid finer out of nowhere: it moves the grid so that none of the sixteen positions is wasted on empty range, and finer spacing follows.

FIG 1Where the sixteen codes land
0.65 of range over 15 intervals-0.043code 00code 10.043code 20.087code 30.13code 40.173code 50.217code 60.26code 70.303code 80.347code 90.39code 100.433code 110.477code 120.52code 130.563code 140.607code 15weight value
The group runs from -0.05 to 0.60. Placing code 0 on -0.0433 and code 15 on 0.6067 spends the whole grid inside the data, which is what the zero-point of 1 bought.

The natural guess, that this is rounding each weight to the nearest of sixteen fixed values, is what fails. The sixteen values are not constants of the format. They are s(q−z)s(q - z) for q=0q = 0 to 1515, and ss and zz are recomputed for every group, so the same real weight in two groups is stored as two different integers and comes back as two different numbers. What the bit width fixes is how many levels there are, not where they are. Where they are is set by the range of the group, which is why one extreme weight in a group can cost every other weight in it three bits of precision.

There is a second job the zero-point does that the spacing argument misses. Because w=0w = 0 maps to q=zq = z exactly, a real zero reproduces as a real zero. Zero padding, masked positions and the zeros a rectifier produces all cost nothing, instead of each being rounded to whatever nearby level the grid happens to have. In integer arithmetic, where these formats actually get used, that is often the reason a zero-point is carried at all.

Where it stops paying

If a group is roughly symmetric about zero, vmin⁡v_{\min} is close to −vmax⁡-v_{\max}, the zero-point comes out near the middle of the code range, and the shifted grid sits almost exactly where a symmetric one would have. You have then paid for a stored number per group and recovered nothing. That split is why weight tensors, which are usually near-symmetric, are commonly mapped with the zero-point fixed at zero, while activations, which a rectifier skews hard to one side, keep theirs. It also costs storage: a group of 128 with a 4-bit scale and a 4-bit zero-point spends 8 bits per 128 weights, or 0.06250.0625 bits per weight, which is nothing next to the 4 bits the weight itself takes.

One caveat on the arithmetic. The zero-point has to be stored as an integer, so rounding it shifts the whole grid by up to half a step. The half-step error bound then becomes a full step for weights at the extreme ends of the range, and the fix is either to store the zero-point at slightly higher precision or to accept that the first and last codes are the ones that suffer.

Worth keeping

  • The step size is the range a group spans divided by one less than the number of codes, so one extreme weight in a group coarsens every other weight in that group.
  • The zero-point shifts the integer grid so a lopsided range uses all of its codes, and it makes a weight of exactly zero reproduce as exactly zero.
  • Error per weight is at most half a step, which is why removing the wasted half of the grid and halving the step halves the damage.

Where this goes next

This question is one step of Making a Model Smaller, a course that works through its subject from the beginning. The page that teaches this part of it is Keeping Fewer Bits, and it reads the whole way through whether or not you have an account.

What an account changes is the practice: problems standing under every idea, marked with a reason rather than a tick, in a copy of the course that is yours.

Read that pageThe whole course

Other questions people ask about this

  1. Why does the memory used by a model keep growing as the conversation gets longer?The model's weights do not grow. It keeps a cache of keys and values for every earlier token, so memory rises with conversation length and with how many conversations run at once.
  2. Why does one outlier weight wreck 4-bit quantization for the whole layer?A single huge value sets the rounding step for the whole layer, so ordinary weights fall below one step and round to zero.
  3. Why does quantization-aware training use fake quantization instead of just quantizing at the end?Fake quantisation lets training see the rounded weights in the forward pass while still updating the exact weights, so the model learns weights that survive rounding instead of being rounded only after training.
  4. Why doesn't pruning half the weights make the model twice as fast?Because the zeros stay in the matrix: they are still stored and still multiplied, so the model is the same size and the same speed as before.
  5. Why does the model return JSON wrapped in extra text when I asked for JSON?A request for JSON does not force the model to return only JSON, so you need to validate the reply and repair it once rather than assume it will parse.
  6. Why do the numbers look good while users keep complaining?Because the score measures a fixed set of cases with a fixed rule, and the users are living through behaviour that the set and the rule never covered.

Read alongside