Why the Gradient Is Computed Backwards

Last timeThe Steepest Direction

Getting a hundred billion partial derivatives costs about as much as one forward pass. That is a theorem about the order you multiply matrices in.

The previous lesson treated the gradient as given. Here it has to be produced.

The question is sharper than it looks: a large model has a hundred billion

parameters, the loss is one number, and training needs the derivative of that

one number with respect to every one of the hundred billion. Finite differences

would need a hundred billion forward passes. Doing it in roughly one is the

result that makes the field possible.

A network is a composition

Strip away the names and a network is a chain. An input goes into the first

operation, its output goes into the second, and so on until a single number

comes out.

FIG 1A short chain, forward and backward

Solid arrows are the computation, dotted arrows are the sweep that follows it. The backward sweep visits each node exactly once, in reverse.

Call the operations $f_1, f_2, \ldots, f_k$ , so the loss is

$L = f_k(f_{k-1}(\cdots f_1(\theta)))$ . The chain rule for vector functions says

the derivative of the composition is the product of the derivatives of the

parts, where each part's derivative is a matrix: the Jacobian, holding the

derivative of every output of that operation with respect to every input.

The lesson stops here

7 more paragraphs to go

You have read the opening. The rest of the argument, the problems that check whether it landed, and the lines worth keeping at the end all come with a plan.

The first lesson of every course in the library reads the whole way through, free, so you can see exactly what the rest of them are.

See the plan The contents

This is the reading half

Starting the course gives you your own copy of it. Every idea on every page has problems standing under it, marked with a reason rather than a tick, and any sentence you do not believe can be opened and argued with. None of that can happen on a page nobody owns.

The contents

Why the Gradient Is Computed Backwards

A network is a composition

7 more paragraphs to go

This is the reading half

Read alongside