Dividing by the Size of Your Own Gradients

Last timeWhy a Noisy Gradient Still Works

Adaptive methods give every parameter its own step size using quantities already in hand. Here is what that buys, what it cannot fix, and why the correction term exists.

The conditioning lesson identified the exact fix and then said it was

unavailable: rescale each direction by its curvature, and the ellipse becomes a

circle. The matrix that would do it has $n^2$ entries and needs inverting.

Adaptive methods take the part of that fix which is affordable. Restrict the

rescaling to the diagonal, so it is one number per parameter rather than a

matrix, and estimate those numbers from the gradients that the run is producing

anyway. Nothing is measured that was not already there, and the memory cost is

two extra numbers per parameter.

The denominator

Keep a running average of the square of each coordinate's gradient, and divide

that coordinate's step by its square root:

v_t = \beta_2 v_{t-1} + (1 - \beta_2) g_t^2, \qquad \theta_{t+1} = \theta_t - \frac{\alpha \, g_t}{\sqrt{v_t} + \epsilon}

The square is taken coordinate by coordinate, so $v$ is a vector the same shape

as the parameters, and each parameter is divided by its own history. A

coordinate whose gradients are routinely large gets a small step; a coordinate

whose gradients are routinely tiny gets a large one.

That is the connection back to conditioning. In the stretched bowl of the

earlier lesson the sharp direction has large gradients and the flat direction

has small ones, so this division pushes both toward the same effective rate. It

is preconditioning with the typical gradient magnitude standing in for the

curvature, which is a rough substitute but a free one.

The invariance is worth checking directly. Multiply the loss by a thousand.

Every gradient becomes a thousand times larger, $v$ becomes a million times

larger, $\sqrt{v}$ becomes a thousand times larger, and the ratio is exactly

what it was. Plain gradient descent would have needed its learning rate divided

by a thousand. This is the property that makes adaptive methods forgiving, and

it is also why their learning rate is not comparable to the learning rate of

plain descent: here $\alpha$ is close to the actual distance moved per step,

because the fraction it multiplies is of size roughly one.

The lesson stops here

7 more paragraphs to go

You have read the opening. The rest of the argument, the problems that check whether it landed, and the lines worth keeping at the end all come with a plan.

The first lesson of every course in the library reads the whole way through, free, so you can see exactly what the rest of them are.

See the plan The contents

This is the reading half

Starting the course gives you your own copy of it. Every idea on every page has problems standing under it, marked with a reason rather than a tick, and any sentence you do not believe can be opened and argued with. None of that can happen on a page nobody owns.

The contents

Dividing by the Size of Your Own Gradients

The denominator

7 more paragraphs to go

This is the reading half

Read alongside