Round It, Store the Integer, Multiply It Back
Last timeWhat Is Actually Taking the Room
Storing a weight in four bits is two multiplications and a rounding. Doing the arithmetic on paper is what makes the failure case obvious before it reaches your model.
The previous lesson ended with a formula that had two factors in it: the number
of parameters, and the bytes each one takes. The second factor is the one you
can change without retraining anything, and this lesson is the whole of how.
The idea is older than neural networks and is almost embarrassingly simple. A
weight is some number like 0.0374. Storing that to full precision takes two or
four bytes. But you do not need its exact value: you need a number close enough
that the model still behaves. So pick a step size, work out how many steps from
zero your weight is, and store that count instead. The count is a small whole
number and fits in a few bits.
- the stored integer, which is all that goes in memory
- the original weight
- the step size, one number shared by a group of weights
The mapping
Work one through by hand. Suppose the step size is 0.01 and the weight is
0.0374. Divide: 3.74. Round: 4. Store the number four, which needs three bits.
To use it, multiply back: 4 times 0.01 is 0.04. The model now computes with
0.04 where it was trained with 0.0374, an error of 0.0026, which is less than
half of the 0.01 step.
That bound is general and it is the useful thing to remember. Rounding to the
nearest multiple of a step can never be wrong by more than half that step,
because if it were, the next multiple along would have been nearer.
What sets the step size
So everything depends on the step. It is not chosen freely: it is forced by two
things, the range you have to cover and the number of distinct values your bits
can express.
The lesson stops here
5 more paragraphs to go
You have read the opening. The rest of the argument, the problems that check whether it landed, and the lines worth keeping at the end all come with a plan.
The first lesson of every course in the library reads the whole way through, free, so you can see exactly what the rest of them are.
See the planThe contentsThis is the reading half
Starting the course gives you your own copy of it. Every idea on every page has problems standing under it, marked with a reason rather than a tick, and any sentence you do not believe can be opened and argued with. None of that can happen on a page nobody owns.
The contents