The Mathematics of Gradient Descent
The Gradient You Use Is Always the Wrong One
Last timeMomentum
A minibatch gradient is an estimate that is wrong at every step. Being unbiased is what saves it, and the noise that remains sets the floor a run settles at.
Everything so far has assumed the gradient was handed over correctly. It is
not. The loss being minimised is an average over a training set that may hold
billions of examples, and computing its exact gradient once would take longer
than most runs take in total. What is used instead is the gradient of a handful
of examples.
So every step of every training run moves in a direction that is not the
direction the previous lessons derived. The question is why that is survivable.
What a minibatch gradient is
The loss is an average, and the gradient of an average is the average of the
gradients. Draw a batch at random and average over it instead.
- the loss on a single training example
- the full training loss, the average over all N examples
- the batch actually drawn, of size much smaller than N
That last equality is the whole licence for the method. It does not say the
batch gradient is close to the true gradient. On a batch of thirty-two it will
usually point somewhere noticeably different. It says the estimate is centred
correctly, so the errors are as likely to be one way as the other.
The difference matters because descent takes thousands of steps. An error that
averages to zero cancels across steps, and the run follows the true gradient in
the aggregate even though no individual step does. A bias, however small, would
never cancel, and would quietly move the destination. This is why sampling
uniformly is not a detail: a batch drawn preferentially from easy examples
gives a biased gradient, and a biased gradient minimises a different function
than the one you wrote down.
The lesson stops here
7 more paragraphs to go
You have read the opening. The rest of the argument, the problems that check whether it landed, and the lines worth keeping at the end all come with a plan.
The first lesson of every course in the library reads the whole way through, free, so you can see exactly what the rest of them are.
See the planThe contentsThis is the reading half
Starting the course gives you your own copy of it. Every idea on every page has problems standing under it, marked with a reason rather than a tick, and any sentence you do not believe can be opened and argued with. None of that can happen on a page nobody owns.
The contents