Why do gradients become tiny in the early layers of a deep network?

The natural guess is that the early layers get small gradients because they are far from the loss, as though the signal weakened gradually with distance, and that the cure is therefore to train longer or raise the step size. The picture is wrong in a way that matters, because distance is not what enters the calculation. The calculation is a product, so the fall is geometric: after forty layers a factor of 0.99 has already cost you a third of the signal, and after eighty it has cost you more than half, with no single layer doing anything unusual. The second common guess is that some particular layer must be at fault, a saturating unit with a tiny slope. It need not be. A per-layer factor of 0.9 requires no layer to be near saturation, and a run made entirely of healthy-looking layers will still show a gradient at the bottom that is four orders of magnitude too small.

What makes this hard to notice is that it does not look like a failure from outside. The upper layers are only a few factors from the loss, so they train normally; the loss falls for a while as they do what they can with whatever the untrained lower layers happen to emit, then flattens at a mediocre value. Nothing in the loss curve announces that the bottom half of the network has not moved since initialisation. An exploding gradient, the same arithmetic with the factor above one, is the opposite: the loss becomes unreadable in a single step, in the first place anyone looks.

Because the gradient is linear in the loss, multiplying the loss by a constant multiplies every gradient in the model by that same constant, which lifts the small ones clear of the floor of the number format. The constant is divided out again before the parameters are updated, so the direction of the step is untouched. A thousandfold lift moves the early-layer values from around $10^{-6}$ to around $10^{-3}$, comfortably inside the range a narrow sixteen-bit format can hold. The headroom at the top is what limits the constant: push it too far and the late-layer gradients, already around $10^{-3}$, overflow. Note that the trick has to be applied before the backward pass, because once a value has rounded to zero in the format, no later multiplication can bring it back.

The argument reverses if the typical factor exceeds one: the same product grows instead of shrinking, and the symptom is an enormous step rather than a stalled one. That case is handled by measuring the overall length of the gradient and, if it is above a threshold, scaling the whole vector down to that threshold. The direction survives exactly, so the step still goes where the gradient pointed, and the relative sizes of different parameters survive too, which is what tells the optimiser which ones are implicated. Scaling each parameter's gradient separately would destroy exactly that information. There is also a third possibility: if every factor is exactly one the product neither grows nor shrinks, which is the target that careful initialisation and normalisation are aiming at. Residual connections change the shape of the product, since $h_\ell = h_{\ell-1} + f(h_{\ell-1})$ gives $\partial h_\ell / \partial h_{\ell-1} = 1 + f'$, so the backward pass becomes a sum over many paths each close to one rather than a single product, and the gradient keeps its size to the bottom.

One further complication undercuts the obvious diagnostic. Most modern optimisers divide each gradient by a running estimate of its own typical magnitude, so a gradient that is uniformly a thousand times too small produces almost the same update as one of the right size: the thousand appears in the numerator and the denominator alike. A scaling problem can therefore be invisible in the loss curve and still be present in the computation. It stops being invisible at the point that matters. Once the values have rounded to zero there is nothing to divide, the running estimate decays to zero with them, and those parameters stop moving altogether. Recording the gradient length at several depths, rather than only at the output, is what separates a vanishing gradient from the other reasons a loss can flatten early.