A Zero Still Occupies Its Place in the Row
Last timeRounding After, or Training With
Removing scattered weights and removing whole rows sound like the same technique. Only one of them makes a model smaller or faster on ordinary hardware.
Removing weights sounds like the same kind of idea as storing them in fewer
bits, and it is not. Fewer bits keeps every weight and describes each one more
crudely. Removal keeps some weights exactly as they were and deletes others.
The important split is inside removal itself, and it is the thing this lesson
exists for. There are two families, they are reported with the same vocabulary
and the same impressive percentages, and one of them does nothing at all to
the memory or the speed of an ordinary served model.
What scattered removal actually produces
Take a block of weights, find the smallest ones by absolute value, and set them
to zero. The usual report is that the layer is now fifty per cent sparse.
| c1 | c2 | c3 | c4 | c5 | c6 | |
|---|---|---|---|---|---|---|
| row 1 | 0.41 | 0.00 | 0.00 | 0.63 | 0.00 | 0.22 |
| row 2 | 0.00 | 0.37 | 0.58 | 0.00 | 0.19 | 0.00 |
| row 3 | 0.26 | 0.00 | 0.00 | 0.00 | 0.44 | 0.31 |
| row 4 | 0.00 | 0.51 | 0.33 | 0.28 | 0.00 | 0.00 |
| row 5 | 0.18 | 0.00 | 0.47 | 0.00 | 0.00 | 0.39 |
| row 6 | 0.00 | 0.29 | 0.00 | 0.36 | 0.55 | 0.00 |
This is the whole objection. A matrix is stored as a dense rectangle of numbers
and multiplied by hardware that works through every position in it. A zero
occupies its bytes like any other value and consumes its multiplication like
any other value. Nothing was saved, because nothing about the shape changed.
Two things can rescue it. A sparse storage format stores only the nonzero
values plus their positions, which saves memory once the sparsity is high
enough to pay for the position bookkeeping, and is usually slower to compute
with. Or hardware with explicit support for a fixed pattern, such as two
nonzeros in every group of four, which does deliver a genuine speedup but
constrains which weights you may remove.
Absent one of those, scattered removal is a research result rather than a
deployment technique, and it is worth knowing which one you are being shown.
The lesson stops here
5 more paragraphs to go
You have read the opening. The rest of the argument, the problems that check whether it landed, and the lines worth keeping at the end all come with a plan.
The first lesson of every course in the library reads the whole way through, free, so you can see exactly what the rest of them are.
See the planThe contentsThis is the reading half
Starting the course gives you your own copy of it. Every idea on every page has problems standing under it, marked with a reason rather than a tick, and any sentence you do not believe can be opened and argued with. None of that can happen on a page nobody owns.
The contents