Basis Lab

Answers · The mathematics underneath

Why does adding a shortcut connection stop gradients from vanishing?

In shortA shortcut gives the gradient a route whose factor is one plus the layer's own factor, so many layers in series no longer multiply the signal down to nothing.

The gradient reaching an early layer is a product with one local factor for every layer between it and the loss. A shortcut connection adds the layer's input to what the layer computes, so each factor becomes one plus the layer's own factor, and the product now contains an identity route whose factor is exactly one at every depth.

hˉ1=Lˉ∏k=2Lfk\bar{h}_{1} = \bar{L}\prod_{k=2}^{L} f_k

Take a plain stack in which every layer contributes the same factor. With a factor of 0.9, the derivative after fifty layers is 0.9500.9^{50} of what entered at the loss; with a factor of 1.1 it is 1.1501.1^{50}. Neither factor is far from one, yet the two outcomes are not close: the first has shrunk to about five thousandths, the second has grown to about a hundred and seventeen. The product is exponential in the number of layers, so an average factor a tenth below one is already catastrophic at depth fifty, and an average factor a tenth above one is already unstable.

0.950=e50ln⁡0.9=e−5.268≈0.00515,1.150=e50ln⁡1.1=e4.766≈117.40.9^{50}=e^{50\ln0.9}=e^{-5.268}\approx0.00515,\qquad 1.1^{50}=e^{50\ln1.1}=e^{4.766}\approx117.4
FIG 1What a product does over depth
0.00100.00200.00300.00400.001.015.830.545.360.0layers between here and the loss
a typical factor of nine tenthsa typical factor of one and a tenth
Both curves start at a factor a tenth away from one. By fifty layers the descending curve is near five thousandths and the ascending one near a hundred and seventeen.

Saturating functions make the small case systematic rather than unlucky. A squashing unit flattens for large inputs, and a flat unit has a derivative near zero. Take a saturated unit whose local factor is 0.01: the forward value looks confident and settled, while the derivative that would move it has almost closed. Put fifty such units in series and the product is 0.01500.01^{50}, which is 10−10010^{-100}. Nothing overflows and nothing raises an error; the early layers simply receive no usable signal while the late layers train normally.

Now change one layer so that it computes hk+1=hk+Fk(hk)h_{k+1}=h_k+F_k(h_k) instead of hk+1=Fk(hk)h_{k+1}=F_k(h_k). The derivative of the output with respect to the input is no longer Fk′(hk)F_k'(h_k) alone: the direct path contributes one. Writing gk=Fk′(hk)g_k=F_k'(h_k), the local factor is 1+gk1+g_k. If the computed branch has collapsed and gk=0g_k=0, the factor is exactly one. If it is saturated at gk=0.01g_k=0.01, the factor is 1.01.

hk+1=hk+Fk(hk),fk=∂hk+1∂hk=1+Fk′(hk)=1+gkh_{k+1}=h_k+F_k(h_k),\qquad f_k=\frac{\partial h_{k+1}}{\partial h_k}=1+F_k'(h_k)=1+g_k

Repeat that saturated factor fifty times. Without the shortcut the product is 0.0150=10−1000.01^{50}=10^{-100}. With the shortcut round each unit it is 1.01501.01^{50}, and 1.0150=e50ln⁡1.01=e0.4975≈1.641.01^{50}=e^{50\ln1.01}=e^{0.4975}\approx1.64. The computed branches still contribute their 0.01, but the one from the shortcut is present in every factor, so the small contributions are no longer the only thing multiplying the signal.

What the shortcut is not

The natural guess is that a shortcut is an accuracy trick: it gives the network an easier function to fit, or it carries information forward, and the gradient improvement is a side effect of better representations. The gradient effect is earlier than that. It is a change in the local factor itself, from gkg_k to 1+gk1+g_k. The identity path is not only a forward-pass copy of the input. It is a term in the derivative that is one at every depth, so even when every computed branch is saturated and contributes almost nothing, the product has a route that contains none of those small factors.

∏k=1L(1+gk)=1+∑k=1Lgk+∑j<kgjgk+⋯+∏k=1Lgk\prod_{k=1}^{L}(1+g_k)=1+\sum_{k=1}^{L}g_k+\sum_{j<k}g_jg_k+\cdots+\prod_{k=1}^{L}g_k

The leading one on the right is the identity route. If every gk=0g_k=0 the whole product is one and the arriving derivative is Lˉ\bar{L}, not zero. The other terms still contain the small factors, and they can be tiny, but they no longer multiply the entire signal. That is why the fix is described as adding a one: it changes the product rather than the representation.

Where the argument stops

The one is not unconditional. If gkg_k is close to −1-1, then 1+gk1+g_k is close to zero, and a long stack of such factors can still vanish; if gkg_k is large positive, the product can explode. A shortcut with a learned scaling or a projection has factor s(1+gk)s(1+g_k) rather than 1+gk1+g_k, and a small ss can put the shrinkage back. What the shortcut removes is the forced collapse of a long chain of small positive factors, not every possible bad product. Normalisation and careful initialisation keep gkg_k near zero so the factors sit near one, and clipping caps the product after it has grown too large. Those act at different points: the shortcut changes the factor, normalisation changes the inputs that produce it, initialisation changes the starting weights, and clipping changes the final number.

Worth keeping

  • With one local factor per layer, the derivative reaching an early layer is a product whose size is exponential in depth.
  • A shortcut makes each local factor one plus the layer's own factor, so the product contains an identity route that does not shrink.
  • A factor close to minus one or a large positive factor can still make the product vanish or explode, so the shortcut is not an unconditional guarantee.

Where this goes next

This question is one step of Backpropagation in Full, a course that works through its subject from the beginning. The page that teaches this part of it is Why the Signal Fades or Explodes, and it reads the whole way through whether or not you have an account.

What an account changes is the practice: problems standing under every idea, marked with a reason rather than a tick, in a copy of the course that is yours.

Read that pageThe whole course

Other questions people ask about this

  1. Why does backpropagation never build the full Jacobian?Backpropagation only needs a vector multiplied by each layer's derivative matrix, so it computes that product directly and never builds the matrix.
  2. Why does softmax followed by cross entropy have gradient p minus y?Because the derivative of the loss through softmax simplifies to the predicted probability minus the target, one number per class, with the exponentials and fractions cancelling out.
  3. Why do derivatives multiply along a chain but add when the path splits?Along a chain each step scales the effect that arrived, so the scalings multiply; at a split each use contributes its own effect to the same change, so the contributions add.
  4. What does a computation graph actually let you differentiate?It lets you differentiate a final number with respect to any earlier value in the graph by multiplying local derivatives along each route and adding across routes.
  5. What is the difference between variance and standard deviation?They measure the same spread: variance is the average squared distance from the mean, standard deviation is its square root, so you report the standard deviation and calculate with the variance.
  6. How do you make a vague claim precise enough to tell whether it's true?Fix what is being talked about, what is asserted, which cases are covered and the cut-off, then name an observation that would prove it false.

Read alongside