Basis Lab

Answers · The mathematics underneath

What is the difference between the range and the precision of a float32?

In shortRange is the span of sizes a float32 can hold before it becomes infinity or zero; precision is how many significant digits it keeps, so nearby values stay distinguishable.

Range is the span of magnitudes a float32 can name: roughly 10−3810^{-38} to 103810^{38} in normal form. Precision is how many significant digits it keeps at any point inside that span: about seven decimal digits, so two values a million apart in size are still told apart only to a relative gap of about one part in ten million. The two are separate because they are spent from separate fields. A float32 uses 32 bits: one for sign, eight for exponent, twenty-three for fraction. The exponent decides where the leading digit sits, and the fraction decides how many digits follow it. A number can be well inside the range and still have its smallest useful increment swallowed, because precision is a relative promise, not an absolute one.

FIG 1The fields of a float32
one bit of sign
twenty-three stored fraction bits after the implicit leading 1
eight stored exponent bits, normally 1 to 254
the bias that makes the exponent range include negative powers
The exponent field sets the magnitude and therefore the range. The fraction field sets the significant digits and therefore the precision. They are independent budgets inside the same 32 bits.

The exponent sets the range

The exponent field is the range budget. It stores ee in eight bits, with bias 127127, so normal values use e=1e=1 through e=254e=254, giving powers 2−1262^{-126} up to 21272^{127}. At the top the significand is 1.111…21.111\ldots_2, which equals 2−2−232-2^{-23}, not 22. Multiplying that by 21272^{127} gives the largest finite float32.

vmax⁡=(2−2−23) 2127≈3.4028235×1038vmin⁡ normal=1.0×2−126≈1.1754944×10−38v_{\max} = (2-2^{-23})\,2^{127} \approx 3.4028235\times10^{38} \qquad v_{\min\,\text{normal}} = 1.0\times2^{-126} \approx 1.1754944\times10^{-38}

The largest finite value is 3.4028235×10383.4028235\times10^{38}. The smallest positive normal value is 1.0×2−126=1.1754944×10−381.0\times2^{-126}=1.1754944\times10^{-38}. The ratio is about 2.895×10762.895\times10^{76}, and log⁡10\log_{10} of that is 76.4676.46, so the normal range covers about 76 orders of magnitude. That is the whole range budget, and it is why an overflow happens at a definite ceiling rather than by gradual loss of digits.

The fraction sets the precision

Precision is the fraction budget. The stored 2323 bits plus the implicit leading 11 make 2424 significant binary bits. Converting to decimal, 24×log⁡102=24×0.30103=7.224724\times\log_{10}2=24\times0.30103=7.2247, so a float32 carries about 7.27.2 decimal digits. Near 11 the gap between neighbouring float32 values is 2−232^{-23}.

2−23≈1.1920929×10−7106×2−23≈0.11920929103×2−23≈0.000119209292^{-23} \approx 1.1920929\times10^{-7} \qquad 10^{6}\times2^{-23} \approx 0.11920929 \qquad 10^{3}\times2^{-23} \approx 0.00011920929

At magnitude 10610^6 the gap is 106×1.1920929×10−7=0.1192092910^6\times1.1920929\times10^{-7}=0.11920929. At 10310^3 it is 0.000119209290.00011920929. The same seven digits are present in both cases; what changes is the size of the unit they are counting. That is why precision is called relative. It does not mean the float32 can resolve 10−710^{-7} everywhere; it means it can resolve about one part in 10710^7 of whatever magnitude the exponent has selected.

FIG 2Precision is a relative gap
0.000.040.070.110.151.0250000.8500000.5750000.31000000.0magnitude of the number
float32 spacing
The gap between neighbouring float32 values is proportional to the magnitude. Near one it is about 1.19e-7. Near a million it is about 0.119, even though the number of significant digits has not changed.

The natural guess is that a number just past the maximum comes back as a slightly wrong large number, as though the format were rounding. It does not. Above 3.4028235×10383.4028235\times10^{38} the result is infinity. Once infinity is in a sum, the sum is infinity; infinity minus infinity is NaN, and NaN then spreads through every operation that touches it. That is a range failure, and it is abrupt. Precision failure looks completely different: at 11 the gap is 1.1920929×10−71.1920929\times10^{-7}, so 1+10−81+10^{-8} evaluates to 11. At 10610^6 the gap is 0.119209290.11920929, so adding 0.010.01 to 10610^6 leaves 10610^6. The second number was nowhere near the maximum, and it still lost the addition.

At the bottom end the behaviour is less abrupt but not more benign. Below the smallest normal value the format enters the subnormal range, where the implicit leading 11 is dropped and the fraction supplies the value at a fixed exponent. The smallest positive subnormal is 2−23×2−126=2−1492^{-23}\times2^{-126}=2^{-149}, about 1.401298×10−451.401298\times10^{-45}. Between 1.1754944×10−381.1754944\times10^{-38} and that value the number of significant bits falls away, and below 1.401298×10−451.401298\times10^{-45} the result is zero. Zero is a perfectly ordinary float32, so arithmetic continues; a gradient that has become zero cannot be told apart from a gradient that is genuinely zero.

Changing the format changes the split, not the distinction. In float16 the exponent is five bits and the fraction ten, so the largest finite value is 6550465504, the smallest normal is 2−14=6.103515625×10−52^{-14}=6.103515625\times10^{-5}, and the significant bits are 1111, giving 11×0.30103=3.31111\times0.30103=3.311 decimal digits. In float32 the corresponding numbers are 3.4028235×10383.4028235\times10^{38}, 1.1754944×10−381.1754944\times10^{-38} and 7.22477.2247 digits. A fixed bit budget can be spent on exponent or fraction, and a format that is generous with one is often miserly with the other. Within a float32, the safe place to work is near one, where the gap is 1.1920929×10−71.1920929\times10^{-7} and neither edge is close.

Worth keeping

  • A float32 has roughly seven significant decimal digits, not seven decimal places.
  • Exceeding the maximum gives infinity; falling below the smallest subnormal gives zero.
  • Precision loss is relative: the gap between neighbouring float32 values grows in proportion to magnitude.

Where this goes next

This question is one step of Keeping The Numbers In Range, a course that works through its subject from the beginning. The page that teaches this part of it is What A Machine Can Hold, and it reads the whole way through whether or not you have an account.

What an account changes is the practice: problems standing under every idea, marked with a reason rather than a tick, in a copy of the course that is yours.

Read that pageThe whole course

Other questions people ask about this

  1. Why does normalisation subtract the mean and divide by the standard deviation?Subtracting the average removes the offset and dividing by the spread removes the size, so the output has zero average and unit spread whatever scale the input had.
  2. Why does adding a layer's input to its output make very deep networks easier to train?Because the addition puts a derivative of exactly one on the path through it, so the gradient reaching the early layers is not a long product of small factors and does not shrink with depth.
  3. Why do gradients become tiny in the early layers of a deep network?Because the backward pass multiplies one number per layer, and a factor slightly below one, repeated over many layers, collapses towards nothing.
  4. What is the difference between variance and standard deviation?They measure the same spread: variance is the average squared distance from the mean, standard deviation is its square root, so you report the standard deviation and calculate with the variance.
  5. How do you make a vague claim precise enough to tell whether it's true?Fix what is being talked about, what is asserted, which cases are covered and the cut-off, then name an observation that would prove it false.
  6. How do you find the parabola that best fits a function near a point?Match the function's height, slope and bending at the chosen point; those three conditions single out exactly one parabola.

Read alongside