Answers · The mathematics underneath

Why do derivatives multiply along a chain but add when the path splits?

In shortAlong a chain each step scales the effect that arrived, so the scalings multiply; at a split each use contributes its own effect to the same change, so the contributions add.

Derivatives multiply along a chain because each operation scales the same small disturbance in turn, and successive scalings compose by multiplication. They add across a split because a small change at the shared value reaches the end by several separate routes, and the total effect is the sum of those routes' effects.

Along a chain

Let $x$ feed $b$ , let $b$ feed $c$ , and let $c$ feed the loss $L$ . A small change $dx$ first changes $b$ by $(\partial b/\partial x)dx$ . The second operation sees that change in $b$ and scales it again, producing $dc = (\partial c/\partial b)db = (\partial c/\partial b)(\partial b/\partial x)dx$ . The final operation scales once more, so $dL = (\partial L/\partial c)dc = (\partial L/\partial c)(\partial c/\partial b)(\partial b/\partial x)dx$ . Divide by $dx$ and the three local factors multiply. Each factor is local: it knows only its own operation, not the rest of the graph.

\frac{\partial L}{\partial x} = \frac{\partial L}{\partial c} \cdot \frac{\partial c}{\partial b} \cdot \frac{\partial b}{\partial x}

Take local factors $0.4$ at $c$ , $2.0$ at $b$ , and $0.25$ at $x$ . Start at the loss with derivative $1$ . The third operation leaves $1 \times 0.4 = 0.4$ ; the second leaves $0.4 \times 2.0 = 0.8$ ; the first leaves $0.8 \times 0.25 = 0.2$ . The same product is $0.4 \times 2.0 \times 0.25 = 0.2$ . A factor below one shrinks the signal, a factor above one magnifies it, and both happen in the same chain. The product can become very small or very large simply because many factors are multiplied; that is arithmetic, not a separate failure mode.

Where the path splits

Let $a$ be used by two operations, one producing $p$ and one producing $q$ , and let both $p$ and $q$ affect $L$ . A small $da$ changes $p$ by $(\partial p/\partial a)da$ and $q$ by $(\partial q/\partial a)da$ . The effect on $L$ through $p$ is $(\partial L/\partial p)(\partial p/\partial a)da$ ; the effect through $q$ is $(\partial L/\partial q)(\partial q/\partial a)da$ . Because these are two first-order changes to the same $L$ , their sum is the first-order change in $L$ . Divide by $da$ and the two terms add.

\frac{\partial L}{\partial a} = \frac{\partial L}{\partial p} \cdot \frac{\partial p}{\partial a} + \frac{\partial L}{\partial q} \cdot \frac{\partial q}{\partial a}

With $\partial L/\partial p = 3$ and $\partial p/\partial a = 0.5$ , the first route contributes $3 \times 0.5 = 1.5$ . With $\partial L/\partial q = 2$ and $\partial q/\partial a = 0.25$ , the second route contributes $2 \times 0.25 = 0.5$ . The total is $1.5 + 0.5 = 2.0$ . If the second term is omitted, the derivative is $1.5$ , too small by $0.5$ .

FIG 1One value, two uses

A change at a reaches L by two separate routes; neither route replaces the other.

The add rule is also why sharing a parameter does not make its derivative smaller. If the same weight is used in several places, each use contributes a term; the gradient is the total of those terms. It is the forward value that was shared, not the derivative. The derivative is accumulated, not divided.

The natural guess is that a split should force a choice: take the largest path, average the paths, or follow one route. That fails because the derivative is a linear approximation. For the same infinitesimal $da$ , both uses change, and neither is selected. The two effects are separate first-order terms, so they add. Multiplication along a chain is not a different rule; it is the same linear approximation applied twice, where the output of one scaling becomes the input of the next. A common hand-written error is to miss a branch term. No error is raised; the gradient is simply too small in that direction.

Where it stops

This is first-order calculus. If the functions are not differentiable, or a branch is a hard switch that does not respond locally, its term is zero rather than a small contribution; for a maximum, the inactive side contributes nothing locally. For a finite change rather than an infinitesimal one, cross terms and curvature appear and the simple sum of first derivatives is no longer exact. It is exact in the limit of small changes, which is the sense in which a derivative is a coefficient.

Worth keeping

Along a chain, the derivative is a product because each operation scales the change that arrives before passing it on.
At a branch, the derivative is a sum because the same change travels through each use separately and all those effects count.
Missing one branch term does not raise an error; it silently makes the gradient too small in that direction.

Where this goes next

This question is one step of Backpropagation in Full, a course that works through its subject from the beginning. The page that teaches this part of it is Composing Derivatives, and it reads the whole way through whether or not you have an account.

What an account changes is the practice: problems standing under every idea, marked with a reason rather than a tick, in a copy of the course that is yours.

Read that page The whole course