Why does adding a layer's input to its output make very deep networks easier to train?

The addition puts a term into the derivative that is exactly one, so the gradient reaching the earliest layers is no longer a long product of small numbers and does not collapse as depth grows. It also makes the identity free: a block only has to output zero from its branch to leave its input untouched, and producing zero is the cheapest thing a layer can do.

Both halves matter, and they are the same fact seen forwards and backwards. Write the block as output equals input plus what the branch computed.

The one is the route around, and it cannot be anything else. The $f'(x)$ is the branch, which may be anything, including nearly zero. Now count. A plain stack of a hundred layers gets its gradient at the bottom by multiplying a hundred numbers together, one factor per layer. If those factors average 0.9, which nobody would notice at a single layer, the product is $0.9^{100}$, which is about 0.000027: roughly three parts in a hundred thousand, and the early layers are effectively frozen while the late ones move. Even a factor of 0.99 per layer gives $0.99^{100} \approx 0.37$, so two thirds of the signal is gone before it reaches the bottom.

With the addition, each block contributes a factor of one plus something. The route that takes no branch at all multiplies by one at every block, so its contribution to the gradient at the bottom is the gradient at the top, unchanged, whatever the hundred branches are doing. Fifty layers in, a plain stack at 0.9 has kept $0.9^{50} \approx 0.005$, about one part in two hundred, while the route around has kept all of it.

The natural guess is that adding the input back is a mild convenience, a way of letting information flow forwards, and that the real work is still done by the branches. That guess predicts the wrong thing: it says the benefit should scale gently and that a skip connection is a small improvement to any deep network. What the arithmetic says is that the benefit is categorical. A product of numbers below one tends to zero, and a sum of one plus something does not, and the difference between those two behaviours is not a matter of degree.

The related mistake is cosmetic. Somebody adds the skip, then puts a normalisation or a fixed rescaling on the route around rather than on the branch, and reintroduces a factor per block. Everything looks fine and the vanishing comes back, because the unmultiplied path is gone. Keep the route around empty: no weights, no scaling, no normalisation on it. If you must shrink something, shrink the branch.

The forward version of the argument starts from an impossibility. A network of fifty-six plain layers had higher error on its own training data than one of twenty, trained the same way, on the same data. That should not happen: the deeper network contains the shallower one, since the extra thirty-six layers could copy their inputs through unchanged and the two would compute the same function. So the deeper one can match the shallower one and cannot be worse, unless the optimiser cannot find the copying setting.

It cannot, or not easily. For a plain layer to copy its input it must learn a weight matrix that is exactly the identity, entry by entry, which is a knife edge to hit with gradient descent. For a block with an addition to copy its input it must produce zero from the branch, and zero is where weights start at initialisation and the easiest value for a layer to reach. The identity stops being a needle to thread and becomes the default.

There is a second consequence that is easy to miss. At every block the signal either goes through the branch or around it, so a stack of $n$ blocks contains $2^n$ distinct routes from input to output. At fifty blocks that is $2^{50}$, over a thousand million million routes. Their lengths are not uniform: most take about half the branches, so by count the long routes dominate.

The gradient is not carried by them equally. Every branch a route takes multiplies in another factor that is typically below one, so the short routes carry most of the gradient and the long ones almost none. A very deep residual network therefore does much of its work at an effective depth of ten or twenty, which is why removing one block from a trained residual network barely changes its output while removing one layer from a plain network destroys it. This is also why the branch should start small: initialise the last layer of each branch at zero, or give it a learned factor starting at zero, and the first forward pass is an almost exact identity, so no warm-up is needed and stacks of a thousand layers train without normalisation.

Two assumptions hold this up. The first is that the branch terms do not conspire to cancel the ones, which by symmetry they do not do by accident, but a deliberately chosen $f'$ of minus one would. The second is that the identity route is left alone. If the branch is initialised at full size the path grows multiplicatively with depth, the early gradients are badly scaled, and the step size has to be raised slowly from near zero to survive the first few thousand updates.

And the addition solves only the part of the problem that comes from repeated multiplication along the depth. The activations still have their own scaling to worry about, which initialisation and normalisation address, and a residual network with badly scaled branches will still diverge. What the route around buys is that depth stops being a product, so the questions that remain are about the size of things rather than about whether the gradient arrives at all.