What do I actually lose by storing a model's weights in 8 bits instead of 16?

Half the bytes, and twice the speed for any model that was waiting on memory rather than on arithmetic. What you actually give up is fineness in the stored numbers and, less obviously, reach: a handful of values that cease to be representable at all.

A 70 billion weight model at 16 bits is 140 GB of weights. The same model at 8 bits is 70 GB. That is the whole of the space saving and it is exactly a factor of two, because 16 bits is 2 bytes per weight and 8 bits is 1 byte per weight, and nothing else about the file changes.

The speed follows from the same number, provided you know which cost dominates. Producing one token means reading every weight once: each is used a single time and then not again until the next token. So bytes per weight is bytes per token. On hardware with 3350 GB/s of memory bandwidth:

140 GB at 3350 GB/s takes 0.0418 s, about 42 ms per token, which is 24 tokens a second. 70 GB at 3350 GB/s takes 0.0209 s, about 21 ms per token, which is 48 tokens a second.

The multiplications are identical in both cases. What changes is how long the units sit idle waiting for an operand to arrive. Nothing about the computation is approximated differently, only the rate at which data is fed into it. That is why the gain comes out so clean, and why it disappears entirely if the job is limited by arithmetic instead of movement.

A stored number is not a number in the mathematical sense. It is a sign, a fraction and an exponent, and the two fields answer two quite different questions.

The fraction decides how finely two nearby values can be told apart, because it fixes how many representable values sit between one power of two and the next. Widen it by one bit and every gap halves. The exponent decides how large and how small a value can be held at all. A format of a fixed total width divides its bits between those two jobs, and moving a bit from one to the other trades fineness against reach at a fixed price.

A scale is not stored per weight, because storing one per weight would defeat the purpose. A group of weights shares a single scale, and that scale must accommodate the group's largest member. So a value near 500 is held to about eight parts in a thousand, which is perfectly adequate. A value of 1 in the same group has no code anywhere near it: dividing by the step gives 1/3.937 = 0.25, and it rounds to zero, or to 3.937, with nothing representable in between. Its relative error is not 0.79 per cent, it is 100 per cent, and the contribution that weight was making vanishes.

That is the fineness failure, and it is survivable. Thousands of these small contributions average out, the outputs drift a little, and it takes hundreds or thousands of cases to detect the drift at all. Which is precisely what makes it dangerous: the usual check, trying a few prompts and seeing that the answers look sensible, cannot see it.

The reach failure is not survivable. A value above the largest exponent the format can express becomes infinite, and an infinity poisons every number it subsequently touches. A value below the smallest becomes zero, which quietly removes a contribution rather than announcing itself. Either way the output is not slightly worse, it is wrong, and often visibly wrong. A half-size format with an exponent range from about 2 raised to -14 up to 2 raised to 15 has ample reach for weights, which cluster near 1, but tight reach for running totals, which can sit twenty or thirty powers of two higher. That asymmetry is why the half-size format that bought reach and gave up fineness beat the one that did the opposite, despite being worse on paper at representing any single number.

Large models have a small number of positions, a few out of thousands, whose values run enormously above everything else, and they recur in the same places rather than at random. That is what makes eight bits tractable rather than hopeless: those few can be given their own scale, or the weights can be split into groups of a few hundred with one scale each, so a single large value spoils only its own neighbourhood. Two bytes a weight costs nothing measurable and remains the default; one byte for the weights, with per-group scales and care over the outliers, is routine; half a byte is where quality visibly moves.

Three assumptions are doing work underneath all of this. Describing a system as running in eight bits is almost always shorthand for the weights only: the running totals are accumulated in a wider format whatever the inputs were, because thousands of small errors summed together stop being small, and the stored conversation and the activations are separate decisions with their own formats. The doubling of speed requires that movement is the limit; on a job limited by arithmetic, halving the bytes buys nothing. And the comparison is against 16 bits, not against an ideal: the weights themselves emerged from an averaging process and never carried three decimal digits of real information to begin with, so the first halving is nearly free and the second is where you start paying.