Derive why gradient descent works, what breaks it, and what every optimiser since is repairing, well enough to read a training curve and know which number to change

The Mathematics of Gradient Descent

A derivation-first walk through training, taking the steepest direction, the largest safe step, momentum, noise and per-coordinate scaling each in turn.

8 lessons, written and corrected before you arrived. Reading them here needs no account. The first reads the whole way through; the others open and then stop. Starting the course gives you your own copy, where every idea has problems standing under it and you can ask about any sentence.

Start reading

01Why the Steepest Direction Is the Negative GradientThe claim that the gradient points the way downhill is a theorem with a proof and a hidden assumption. Both are worth having in full.
02Why the Gradient Is Computed Backwardsopening onlyGetting a hundred billion partial derivatives costs about as much as one forward pass. That is a theorem about the order you multiply matrices in.
03The Largest Step a Curvature Will Tolerateopening onlyOn a quadratic the learning rate has an exact ceiling, and the loss curve of a run that exceeds it looks nothing like a run that is merely slow.
04One Ratio Decides How Long Training Takesopening onlyWhen curvature differs between directions, the step size is set by the sharpest and the progress is made in the flattest. The gap between them is the whole cost.
05What Momentum Multiplies Your Step Size Byopening onlyMomentum is a geometric sum of past gradients. Working out what that sum converges to explains both why it helps and why it needs a smaller learning rate.
06The Gradient You Use Is Always the Wrong Oneopening onlyA minibatch gradient is an estimate that is wrong at every step. Being unbiased is what saves it, and the noise that remains sets the floor a run settles at.
07Dividing by the Size of Your Own Gradientsopening onlyAdaptive methods give every parameter its own step size using quantities already in hand. Here is what that buys, what it cannot fix, and why the correction term exists.
08The Variance That Keeps a Deep Network Aliveopening onlyInitial weights have one job: keep the scale of the signal steady through every layer. Being a few percent off compounds into a factor of hundreds by layer fifty.