Two Ways to Accumulate, and Only One Is Affordable
Last timeComposing Derivatives
Derivatives can be accumulated from the inputs forwards or from the loss backwards, and the two compute different things. With millions of inputs and one output, the choice is not close.
There are two ways to apply the chain rule over a graph, and they are not two
implementations of the same thing. They compute different quantities, and which
one you want depends on the shape of the problem rather than on taste.
Forward: pick an input and follow it
Choose one input. Declare its derivative with respect to itself to be one, and
every other input's to be zero. Then run the computation as usual, and alongside
each value carry the derivative of that value with respect to your chosen input.
When the computation ends you know how that one input affects everything,
including the loss.
It is elegant, needs no history, and is useless here. To learn what all the
weights do you would run it once per weight.
Backward: pick an output and trace it back
Now the other direction. Run the forward pass normally. Then declare the
derivative of the loss with respect to itself to be one, and sweep back through
the graph, at each operation turning the derivative at its output into
derivatives at its inputs. When the sweep reaches the start, every value in the
graph, including every weight, has its derivative with respect to the loss.
The lesson stops here
3 more paragraphs to go
You have read the opening. The rest of the argument, the problems that check whether it landed, and the lines worth keeping at the end all come with a plan.
The first lesson of every course in the library reads the whole way through, free, so you can see exactly what the rest of them are.
See the planThe contentsThis is the reading half
Starting the course gives you your own copy of it. Every idea on every page has problems standing under it, marked with a reason rather than a tick, and any sentence you do not believe can be opened and argued with. None of that can happen on a page nobody owns.
The contents