Paying for Precision
Last timeArithmetic and Movement
Every weight is stored in some number format, and the choice of format sets both how much memory the model occupies and how fast it can be read.
Nothing in the previous lesson said how big a weight is. It said two bytes, in
passing, and then used that figure in every sum. This lesson is about where it
came from and what happens when it changes, because it is the one quantity in
the whole course that can be altered by a line of configuration and that moves
every other number at once.
What a number format is spending its bits on
A stored number is not a number in the mathematical sense. It is a fraction
multiplied by a power of two, with both the fraction and the power written out in
a fixed number of bits.
- one bit for the sign, positive or negative
- the fraction, held in a fixed number of bits, which decides how finely two nearby values can be told apart
- the power of two, held in its own fixed number of bits, which decides how large and how small a value can be held at all
- the value that comes out, which is only ever one of the finitely many combinations the two fields allow
Those two fields answer two different questions and they fail in two different
ways. The power of two gives the format its reach: how far above one and how far
below one it can go before it simply has no representation. The fraction gives
the format its fineness: how many distinct values sit between one power of two
and the next. Widen the fraction by one bit and every gap halves.
The lesson stops here
6 more paragraphs to go
You have read the opening. The rest of the argument, the problems that check whether it landed, and the lines worth keeping at the end all come with a plan.
The first lesson of every course in the library reads the whole way through, free, so you can see exactly what the rest of them are.
See the planThe contentsThis is the reading half
Starting the course gives you your own copy of it. Every idea on every page has problems standing under it, marked with a reason rather than a tick, and any sentence you do not believe can be opened and argued with. None of that can happen on a page nobody owns.
The contents