What does a computation graph actually let you differentiate?

A computation graph lets you differentiate a final scalar with respect to any earlier value that actually reaches it, provided every operation on the way has a derivative. The derivative is not stored on a node. Each operation supplies a local derivative, and the graph supplies the routes along which those local derivatives are multiplied and added.

The formula is the chain rule written for a graph. Along one route from $x$ to $L$ you multiply the local derivatives of the operations you pass through, because each small change is fed into the next. If there is more than one route, the changes happen at the same time and their effects add. The product along a route is the derivative of the loss with respect to the start of that route, holding the other route fixed; the sum removes that holding fixed. That is why the graph is recorded with edges rather than as a list of layers: an edge says this value was used here, and a second edge from the same value says it was used again somewhere else.

Take the unit $x=2$, $w=0.5$, $b=0.3$, with $u=wx$, $v=u+b$, $y=\tanh(v)$ and loss $L=y$. Then $u=0.5\times2=1$, $v=1+0.3=1.3$, and $y=\tanh(1.3)\approx0.862$. The local derivative of the squash is $dy/dv=1-y^2\approx1-0.862^2=0.257$. The add has local derivative $1$ with respect to either input. The multiply has local derivative $x=2$ with respect to $w$, and $w=0.5$ with respect to $x$.

At the multiply node the local derivatives are $du/dw=x=2$ and $du/dx=w=0.5$. It is tempting to read $\partial L/\partial w=2$ off that node. That is wrong by the factor $0.257$ that the loss sends back through the squash and the add. The node knows how $u$ changes when $w$ changes. It does not know how $L$ changes when $u$ changes, and that second fact arrives from the other side. The correct value is $0.514$, not $2$.

With $a=2$, the two branches give $p=3\times2=6$ and $q=5\times2=10$, so $r=p+q=16$ and $L=r$. Changing $a$ changes $L$ through $p$ and through $q$. The $p$ route contributes $1\times3=3$. The $q$ route contributes $1\times5=5$. Both happen at the same time, so the total is $3+5=8$. If only one branch were present the derivative would be $3$ or $5$; the second branch is not a correction to the first, it is an additional term. In reverse mode the $1$ from $L$ reaches $p$ and $q$, each sends its local derivative back to $a$, and the two arrivals are added at $a$.

The graph differentiates values, not names. A weight, a bias, an input, and an intermediate activation are all the same kind of object as far as the graph is concerned: each is a value with edges leaving it. A weight used in three places has three outgoing edges, so its derivative is the sum of three contributions, one per use. If a value is used once, its incoming derivative is simply what the next operation sends; if it is used many times, the incoming contributions are summed before the local derivative is applied. Reverse mode starts with $dL/dL=1$ and sweeps once in the reverse of the recorded order, so one backward pass gives the derivative of the same scalar $L$ with respect to every value in the graph. That is why the cost is proportional to the number of edges, not to the number of values squared.

The chain rule on a graph is only as good as the local derivatives. If an operation has no derivative at the value it actually received, for example an absolute value at zero or a hard threshold, no local derivative exists and the graph cannot supply one. A value with no path to the loss has derivative zero, not undefined, because the loss does not depend on it through the recorded graph. The graph is the record of the operations that ran, so a data-dependent branch contributes the paths it actually took. For a value used many times, enumerating paths is correct but exponential. One reverse sweep accumulates the same sum without listing them.