Answers · The mathematics underneath

Why does computing the gradient cost about the same no matter how many parameters the network has?

In shortBecause one backward sweep shares intermediate derivatives across all parameters, so the whole gradient costs a few forward passes in total rather than one per parameter.

The gradient costs about the same because backpropagation does not compute a separate derivative for each parameter. It makes one forward pass to keep the intermediate values and one backward sweep that carries a single row vector of derivatives from the loss back through every operation; the parameter derivatives fall out along the way. The cost is proportional to the number of operations and the size of the activations, not to the number of parameters.

\frac{\partial L}{\partial \theta} = J_k J_{k-1} \cdots J_2 J_1

The product is bracketed from the loss end

The chain rule fixes the factors in that product but not the order in which they are multiplied. Matrix multiplication is associative, so the answer is the same either way, but the cost is not. Multiplying from the right, starting with $J_2 J_1$ , means multiplying a matrix by a matrix, and the partial product stays wide because it has one column per parameter. Multiplying from the left, starting with $J_k J_{k-1}$ , starts with a single row, because the loss is one number. A row vector times a matrix is a row vector, so every later product is again a single row. The chain never builds a matrix with one column per parameter. This is reverse mode, and it gives the derivative of one output with respect to all inputs in one sweep. Forward mode gives the derivative of all outputs with respect to one input per sweep, which is the wrong way round when there is one loss and billions of parameters.

\left(\cdots((J_k J_{k-1})J_{k-2})\cdots\right) \quad\text{versus}\quad \left(\cdots J_3(J_2 J_1)\cdots\right)

FIG 1

Method	Passes needed	Cost relative to one forward pass
Finite differences	one per parameter plus one	about
Forward mode	one per parameter	about
Reverse mode	one forward, one backward	a small constant

A hand check

Take $x = 1$ , $w_1 = 0.5$ , $w_2 = 2$ and target $y = 1$ , with the chain $a = w_1 x$ , $h = \tanh a$ , $p = w_2 h$ , $L = (p - y)^2$ . Forward: $a = 0.5$ , $h = \tanh(0.5) = 0.462117$ , $p = 2h = 0.924234$ , and $L = (0.924234 - 1)^2 = 0.005740$ . Those four numbers are all the forward pass has to produce, and all four are kept.

The backward sweep starts at $L$ with $dL/dL = 1$ . At $p$ , $dL/dp = 2(p - y) = 2(-0.075766) = -0.151532$ . At $h$ , $dL/dh = (dL/dp) w_2 = -0.151532 \times 2 = -0.303064$ . At $a$ , $1 - h^2 = 1 - 0.462117^2 = 0.786448$ , so $dL/da = -0.303064 \times 0.786448 = -0.238320$ . The parameter derivatives then fall out: $dL/dw_2 = (dL/dp)h = -0.151532 \times 0.462117 = -0.070003$ , and $dL/dw_1 = (dL/da)x = -0.238320 \times 1 = -0.238320$ . Each step is one multiplication by a value already computed on the forward pass. No extra pass is made for $w_1$ or $w_2$ .

What the guess misses

The natural guess is that $n$ parameters need $n$ derivatives, so the cost should grow with $n$ . Computing each derivative independently would indeed cost about $n$ forward passes by finite differences, or about $n$ sweeps by forward mode. The mistake is to treat the derivatives as separate objects. The derivative at $h$ is reused on the way to both $w_2$ and $w_1$ , and the derivative at $p$ is reused before that. The chain rule lets the same backward sweep supply every partial derivative, because all of them are entries of one row vector travelling back through the same operations. The row vector grows longer as the parameter count grows, but it is still one vector, and it is still produced by one sweep.

The price of that saving is memory rather than arithmetic. The local derivative of an operation depends on the values that went into it: the slope of $\tanh$ at $a$ needs $h$ , and the derivative with respect to $w_2$ needs $h$ again. So the forward pass cannot discard its activations. Every intermediate value for every layer, and for every example in the batch, stays resident until the backward sweep has passed it. That is usually the binding constraint: the arithmetic of a backward pass is about twice the forward pass, while the memory is roughly the size of every intermediate value the model produced. Checkpointing trades some of that memory back for extra compute by throwing activations away and recomputing them during the sweep. The argument also depends on the loss being a single number. If there were many outputs and few inputs, reverse mode would no longer be the cheap direction, and forward mode would win instead.

The number of parameters changes the length of the row vector, not the number of sweeps. The cost is set by the number of operations and the size of the activations.

Worth keeping

The gradient is a product of Jacobians, and the order of multiplication decides whether the cost grows with the parameter count.
Multiplying from the loss end keeps every intermediate a row vector, so one backward sweep produces every partial derivative.
The saving is paid for in memory, because each operation's local derivative needs the forward values that went into it.

Where this goes next

This question is one step of The Mathematics of Gradient Descent, a course that works through its subject from the beginning. The page that teaches this part of it is Where the Gradient Comes From, and it reads the whole way through whether or not you have an account.

What an account changes is the practice: problems standing under every idea, marked with a reason rather than a tick, in a copy of the course that is yours.

Read that page The whole course