Predict what a machine will do to your arithmetic before it does it, deriving wraparound, the gaps between representable values, rounding error and the catastrophic subtraction from the bits themselves

What a Computer Does to a Number

Integers that wrap, decimals that cannot be stored, and error that grows, all derived from the fixed number of bits a value is kept in.

8 lessons, written and corrected before you arrived. Reading them here needs no account. The first reads the whole way through; the others open and then stop, because a page nobody owns cannot tell who is reading it. Starting the course gives you your own copy, where every idea has problems standing under it and you can ask about any sentence.

Start reading

01Why Eight Bits Hold 256 Values and Minus 128 Has No Positive TwinEight bits are 256 patterns, and a rule decides what they mean. Here are the two rules for whole numbers, the ranges they give, and why one value cannot be negated.
02What Really Happens When a Sum Does Not Fitopening onlyAdding one to the largest value does not raise an error, it lands at the bottom of the range. Here is the carry that does it, the modulus behind it, and where it hides.
03Reading a Real Number Out of Its Bits, and Writing One Tenth Back Inopening onlyThe sign, exponent and fraction fields of a 32 bit and a 64 bit real, decoded by hand from one bit pattern, then a tenth normalised, expanded in binary, rounded, and quoted exactly.
04The Gap to the Next Representable Number, and How to Compute Itopening onlyRepresentable values are evenly spaced between one power of two and the next, so the gap is two to the exponent minus 52, which is derived here and then used on a tenth and on a counter that stops.
05One Rounding Is Half a Gap, and a Tenth Plus Two Tenths Is Exactly a Tieopening onlyThe error of a single floating point operation is at most half the gap at its result, one part in two to the 53, and a tenth plus two tenths lands exactly on a midpoint.
06Error in a Long Sum Grows Like Root n, Not Like nopening onlyThe bound on a sum of a million values allows 999,999 unit roundoffs of error, the measured error is 169, and they differ because the roundings have mixed signs unless something makes them agree.
07The Subtraction That Loses Every Correct Digit at Onceopening onlyA difference between near-equal values magnifies the error already inside them by the ratio of their size to the difference, and two standard formulas are rewritten so it cannot happen.
08Two Zeros, Two Infinities, and a Value Not Equal to Itselfopening onlyWhere the signed zeros, the infinities, the not-a-number values and the subnormals come from, what every comparison does with them, and what a sort and a lookup table do when one arrives.