Follow a derivative all the way from a scalar loss back to every weight in a deep network, with the rule for each kind of operation, the memory the method costs, and the reasons the signal can arrive too small or too large

Backpropagation in Full

One number at the end, millions of weights at the start, and a single sweep that connects them. This course derives backpropagation from the chain rule, works out the rule for each layer, and shows what it costs in memory.

8 lessons, written and corrected before you arrived. Reading them here needs no account. The first reads the whole way through; the others open and then stop. Starting the course gives you your own copy, where every idea has problems standing under it and you can ask about any sentence.

Start reading

01There Are No Layers, Only OperationsBefore any derivative can be taken, the computation has to be broken into pieces small enough that each one has a derivative you already know. What is left is a graph.
02Multiply Along, Add Acrossopening onlyTwo rules cover everything. Along a chain the derivatives multiply, and where a value was used more than once the contributions add. The rest is bookkeeping about the order.
03Two Ways to Accumulate, and Only One Is Affordableopening onlyDerivatives can be accumulated from the inputs forwards or from the loss backwards, and the two compute different things. With millions of inputs and one output, the choice is not close.
04Four Rules That Cover Most of a Networkopening onlyAddition copies the derivative to both inputs, multiplication swaps the two, a matrix product transposes, and an elementwise function scales. Almost every layer in use is built from these four.
05The Matrix Everybody Writes Down and Nobody Buildsopening onlyThe backward step is usually written as a vector meeting a matrix of partial derivatives. That matrix is never built, because only its product with a vector is ever needed.
06Why the Last Two Steps Are One Stepopening onlyThe scores become probabilities and the probabilities become a loss. Doing the backward step through both at once gives an answer so simple that implementations refuse to separate them.
07What the Forward Pass Has to Leave Behindopening onlyEvery backward rule needs some forward value to work with, and those values have to be held from the moment they are produced until the backward pass reaches them. That is the real cost of training.
08A Product of Many Small Numbersopening onlyThe derivative reaching an early layer is a product of one factor per layer in between. Products of many numbers go to zero or to infinity, and almost every architectural trick is a response to that.