There Are No Layers, Only Operations
Before any derivative can be taken, the computation has to be broken into pieces small enough that each one has a derivative you already know. What is left is a graph.
Everyone learns the chain rule twice: once in a calculus class, where the
functions are named and the composition is two deep, and once here, where the
composition is ten million deep and nothing is named at all. The second time is
easier, but only after the computation has been written down in the right way.
Small enough to differentiate
Take a single unit: multiply an input by a weight, add a bias, squash the
result. That is three operations, not one, and the whole point of splitting them
is that each has a derivative you already know.
The choice of pieces is not sacred. A framework might treat the whole multiply
and add as one operation because it is faster to do so, and it would then need
that combined operation's derivative. What cannot change is that at some level
the pieces have to be small enough to differentiate directly.
Edges, and what it means when one splits
An edge means a value was used. A value used by two operations has two edges
leaving it, and that will matter shortly.
A derivative is a question about paths
Here is the whole subject in one sentence, stated before any of it is derived:
the effect of an early value on the final number is a product along each route
and a sum across routes.
| step | value produced | inputs used | must be kept for the backward pass | what happened |
|---|---|---|---|---|
| 1 | 2 | 0 | 1 | The input, x equals 2. Kept, because the derivative with respect to the weight will need it. |
| 2 | 0.5 | 0 | 1 | The weight, w equals 0.5. Kept, because the derivative with respect to the input will need it. |
| 3 | 1 | 2 | 0 | The multiply, u equals 1. The product itself is not needed later, only its inputs. |
| 4 | 1.3 | 2 | 0 | The add, v equals 1.3 with a bias of 0.3. Addition needs nothing kept at all. |
| 5 | 0.862 | 1 | 1 | The squash, y equals about 0.862. Kept, because the derivative of this function is cheapest to write in terms of its own output. |
The order takes care of itself
Nothing in the graph points backwards, and no operation can run before its
inputs exist. That is enough to guarantee an order in which everything is ready
when its turn comes, and running that order is the forward pass. Reverse the
same order and every operation finds the results it needs waiting for it, which
is the backward pass. No cleverness is involved in either.
The graph is a recording
One last thing that surprises people: the graph is not worked out ahead of time
by inspecting the program. It is recorded as the program runs. Each operation,
as it executes, writes down what it did and stashes the values it will need
later. If the program contains a loop whose length depends on the data, the
recording simply comes out a different length that time.
That is why a model whose shape changes with its input is no harder to
differentiate than a fixed one, and why the backward pass never reads any code.
By the time it runs, the code has finished and all that is left is the record.
What to hold on to
Differentiation does not see layers. It sees a graph of small operations, edges
showing which value went where, and a structure in which a derivative is a
product along each path and a sum across paths. The graph is recorded while the
forward pass runs, together with whatever each operation will need on the way
back.
Recap
- A network is not a stack of layers as far as differentiation is concerned; it is a graph of elementary operations, each with two or three inputs and a derivative anyone can write down.
- The graph records which value was used by which operation, and that record is the only thing the backward pass consults: the code that built it is irrelevant by then.
- A derivative of the final number with respect to an early one is a question about every path between them through this graph, which is why the structure matters more than the names of the layers.
This is the reading half
Starting the course gives you your own copy of it. Every idea on every page has problems standing under it, marked with a reason rather than a tick, and any sentence you do not believe can be opened and argued with. None of that can happen on a page nobody owns.
The contents