Answers · The mathematics underneath
Why does normalisation subtract the mean and divide by the standard deviation?
In shortSubtracting the average removes the offset and dividing by the spread removes the size, so the output has zero average and unit spread whatever scale the input had.
Subtracting the average fixes where the numbers sit; dividing by the spread fixes how wide they are. The two are not the same job done twice, and together they leave a collection with an average of zero and a spread of one no matter what scale it arrived on.
A shift slides a collection along the number line and leaves its width alone. A rescale stretches or shrinks its width and leaves its shape alone. Normalisation does one of each, and the order matters, because the spread is measured as a distance from the average. Until the average is out of the way, the number you divide by is partly describing where the collection sits rather than how wide it is.
Take five values arriving at one unit: 2, 3, 4, 5, 6. Their average is 20 divided by 5, which is 4. Subtract it and the values become -2, -1, 0, 1, 2, and their average is zero by construction, since deviations from an average always sum to zero. Squaring those deviations and adding gives 4 + 1 + 0 + 1 + 4 = 10. Dividing by the count of 5 gives a variance of 2, and the square root of that is 1.41. Dividing each centred value by 1.41 gives the final column.
| value | value minus average | squared deviation | divided by 1.41 |
|---|---|---|---|
| 2 | -2 | 4 | -1.41 |
| 3 | -1 | 1 | -0.71 |
| 4 | 0 | 0 | 0.00 |
| 5 | 1 | 1 | 0.71 |
| 6 | 2 | 4 | 1.41 |
Now suppose training doubles every weight in the layer that produced those values. What arrives becomes 4, 6, 8, 10, 12. The average doubles to 8 and the deviations are -4, -2, 0, 2, 4, which are the old deviations doubled. The sum of squares is 16 + 4 + 0 + 4 + 16 = 40, the variance is 8, and the spread is 2.83, exactly twice what it was. Dividing each doubled deviation by the doubled spread cancels the factor of two and returns -1.41, -0.71, 0, 0.71, 1.41, the same five numbers as before. The unit now responds to different things, because its weights have changed, but the size of what it sends on has not changed at all.
The half that looks like housekeeping
The usual guess is that the subtraction is tidying up and the division is the part doing the work, so dividing by the spread on its own ought to be enough. It is not, because dividing fixes one property and leaves the other exactly where it was. Divide the five original values by their spread of 1.41 and you get 1.41, 2.12, 2.83, 3.54, 4.24: the spread is now one, but the average is 4 divided by 1.41, which is 2.83. Push those into a rectifier and every one of them is positive, so the unit is stuck on and carries a constant offset into every example. Do it the other way, subtract the average only, and you get -2, -1, 0, 1, 2: average zero, but a spread of 1.41 that still scales with whatever the layer below happened to produce. Either half alone leaves the input scale visible in the output. Only both together hide it.
The second guess is that the result is confined to a fixed interval such as zero to one, which is what normalising means in other contexts. It is not confined at all, and the arithmetic shows how far it can go. Take 2, 3, 4, 5, 26. The average is 40/5 = 8. The squared deviations are 36, 25, 16, 9 and 324, summing to 410, so the variance is 82 and the spread is 9.06. The normalised values are about -0.66, -0.55, -0.44, -0.33 and 1.99, and the last one alone shows there is no unit interval here. What is true is a bound: with five values the largest possible normalised magnitude is the square root of 4, which is 2, and 1.99 is just inside it. The guarantee is about the average and the spread of the whole collection, not about any single value's range.
Imposing zero and one is a restriction, and some units genuinely need a large output or a deliberate offset, particularly when the nonlinearity that follows is not symmetric about zero. So the step ends with a learned multiplier and a learned offset, one pair per channel, applied after the fixed part. Those two numbers can restore any scale and centre the unit wants, including exactly undoing the normalisation. What has changed is who controls the size. It used to be an accident of every weight in the layer; it is now a single parameter that the optimiser can see and move directly.
Where it stops being true
The formula is exact arithmetic only when the collection has some variation. A collection of identical values has a spread of zero, so a small constant is added under the square root to keep the division defined, and the output is then near zero rather than undefined. That constant is the one approximate thing in the expression, and making it large would soften the normalisation. The learned multiplier and offset are a second escape hatch: a unit that finds the constraint unhelpful can learn a scale close to zero and a large offset and behave as though the step were barely there. Finally, the average has to be taken over some collection, and the size of that choice matters. Averaging over the examples in a batch means one example's output depends on the others in the batch, which requires running estimates to be kept so that a single example can be handled on its own later.
Worth keeping
- Subtracting the average and dividing by the spread are separate operations with separate effects: one moves the collection, the other changes its width.
- Because both the average and the spread of a layer's outputs scale with its weights, dividing one by the other cancels the weight scale entirely.
- Normalisation fixes the average and spread of a collection, not the range of individual values, and a learned scale and shift are applied afterwards to give back the freedom it removed.
Where this goes next
This question is one step of Keeping The Numbers In Range, a course that works through its subject from the beginning. The page that teaches this part of it is Resetting The Scale Partway, and it reads the whole way through whether or not you have an account.
What an account changes is the practice: problems standing under every idea, marked with a reason rather than a tick, in a copy of the course that is yours.
Read that pageThe whole courseOther questions people ask about this
- What is the difference between the range and the precision of a float32?Range is the span of sizes a float32 can hold before it becomes infinity or zero; precision is how many significant digits it keeps, so nearby values stay distinguishable.
- Why does adding a layer's input to its output make very deep networks easier to train?Because the addition puts a derivative of exactly one on the path through it, so the gradient reaching the early layers is not a long product of small factors and does not shrink with depth.
- Why do gradients become tiny in the early layers of a deep network?Because the backward pass multiplies one number per layer, and a factor slightly below one, repeated over many layers, collapses towards nothing.
- What is the difference between variance and standard deviation?They measure the same spread: variance is the average squared distance from the mean, standard deviation is its square root, so you report the standard deviation and calculate with the variance.
- How do you make a vague claim precise enough to tell whether it's true?Fix what is being talked about, what is asserted, which cases are covered and the cut-off, then name an observation that would prove it false.
- How do you find the parabola that best fits a function near a point?Match the function's height, slope and bending at the chosen point; those three conditions single out exactly one parabola.