Answers · The mathematics underneath

Why does backpropagation never build the full Jacobian?

In shortBackpropagation only needs a vector multiplied by each layer's derivative matrix, so it computes that product directly and never builds the matrix.

Because the backward step only ever needs a vector multiplied by the Jacobian, never the matrix by itself. A derivative vector arrives at the output side, meets the Jacobian, and leaves as a derivative vector the size of the input. Every operation supplies a rule for that product, so the matrix is a way of describing the rule rather than something allocated in memory.

\bar{x} = \bar{u}\,J

The size of the object

Take a layer with $n$ inputs and $n$ outputs. Its Jacobian $J$ has one row per output and one column per input, so it has $n^2$ entries. At $n=512$ , that is $512^2=262{,}144$ entries. At $n=4096$ , it is $4096^2=16{,}777{,}216$ . At $n=16384$ , it is $16384^2=268{,}435{,}456$ . Relative to the 512 case, those are 64 times and 1024 times larger, because $(4096/512)^2=8^2=64$ and $(16384/512)^2=32^2=1024$ . The growth is quadratic in width, so doubling the width quadruples the matrix.

FIG 1

width	entries in the matrix	size relative to the small case
512	262144	1
4096	16777216	64
16384	268435456	1024

A batch makes the storage worse before it makes anything better. At width 4096 and a batch of 32, storing one Jacobian per example would need $16{,}777{,}216 \times 32 = 536{,}870{,}912$ entries for one layer. Those entries would be read once and discarded. A smaller batch does not fix the shape: even one example already has as many entries as the dense layer has weights, and several layers multiply the problem. The backward pass does not need them in that form: it needs one vector on the output side and produces one vector on the input side.

FIG 2The backward step, written without building the matrix

: the derivative arriving at the output, one number per output of this operation
: the matrix of every output differentiated against every input, which appears here and is never constructed

The result is the size of the input, not of the matrix. Every operation supplies a rule that produces it directly.

Why the product is enough

For a dense linear layer $y=Wx$ , the Jacobian with respect to $x$ is $W$ itself. Backpropagation needs $\bar{x}=W^\mathsf{T}\bar{u}$ , whose $i$ th entry is $\sum_j W_{ji}\bar{u}_j$ . At width 512, one entry costs 512 multiplications and additions, and all 512 entries cost $512 \times 512 = 262{,}144$ multiply-adds. That is the same count as the forward pass, and it uses the weights already stored. Forming a separate Jacobian would copy those 262,144 numbers first and then multiply, doubling both work and memory for no new information. The matrix is the weights; the backward rule reads them transposed.

For an elementwise operation the saving is starker. If $x_i=\tanh(z_i)$ , the Jacobian is diagonal, with entry $1-x_i^2$ on the diagonal and zero everywhere else. At width 4096 it has $4096^2=16{,}777{,}216$ entries, of which $16{,}777{,}216-4096=16{,}773{,}120$ are zeros. The product $\bar{z}_i=(1-x_i^2)\bar{x}_i$ needs 4096 multiplications. Building the matrix would write 16,777,216 numbers to save 4096. The rule is the product, written for the diagonal case, and the zeros are never created.

The natural guess

The natural guess is that each layer must assemble its Jacobian and then multiply, because the chain rule is written with a matrix. That guess fails on the arithmetic: the matrix has as many entries as the product of the input and output counts, and for ordinary widths that is thousands of times the width of the layer. It also fails on use: each entry is consumed once. A matrix that is written, read once, and thrown away is not a data structure the method needs. The chain rule asks for the product, and the product is what the backward rule returns. Writing the matrix first is a way of explaining the product, not a step in computing it.

Backpropagation gives one vector-Jacobian product per backward pass, not the full Jacobian. If a calculation genuinely needs every entry, for example a complete sensitivity table of every output against every input, then it can be obtained by running one backward pass per output direction. At width 4096 that means 4096 passes, each producing one row of 4096 numbers, for $4096 \times 4096 = 16{,}777{,}216$ entries in total. The cost is real and it is paid only when the full table is the thing wanted. Ordinary training never wants it, because the loss is scalar and the gradient is one vector.

Second-order methods need more than the gradient, but not by materialising a second-derivative matrix. A Hessian-vector product can be obtained by differentiating the backward pass again, so the same rule applies one level up. The cost is a few passes rather than one, which is expensive but possible, whereas storing a full Hessian for a large layer is not possible at any useful size. The gradient calculation itself remains a sequence of vector-Jacobian products, and the Jacobian remains a description. What changes is how many such products are asked for, not whether the matrix is built.

Worth keeping

The backward pass needs a vector-Jacobian product, not the Jacobian, and every operation has a rule that computes that product directly.
For a layer of width n, the full Jacobian has n squared entries, which at n=4096 is 16,777,216 entries consumed once.
If the full Jacobian is genuinely required, it costs one backward pass per output direction, not one pass total.

Where this goes next

This question is one step of Backpropagation in Full, a course that works through its subject from the beginning. The page that teaches this part of it is The Matrix That Is Never Built, and it reads the whole way through whether or not you have an account.

What an account changes is the practice: problems standing under every idea, marked with a reason rather than a tick, in a copy of the course that is yours.

Read that page The whole course