One Ratio Decides How Long Training Takes

Last timeHow Big a Step

When curvature differs between directions, the step size is set by the sharpest and the progress is made in the flattest. The gap between them is the whole cost.

The last lesson ended on an awkward fact. The step size is capped by the

sharpest direction in the model, and every other direction has to live with

that cap. This lesson works out the price, and it turns out to be a single

number that predicts the length of a training run.

Two directions at once

Take a quadratic in two variables whose axes are already lined up with the

coordinates:

f(x_1, x_2) = \tfrac{1}{2}\left(\lambda_1 x_1^2 + \lambda_2 x_2^2\right)

This loses nothing. Any quadratic can be rotated into this form, with the

$\lambda_i$ being the eigenvalues of its Hessian, and gradient descent is

unaffected by the rotation. Call the largest $L$ and the smallest $\mu$ .

The gradient is $(\lambda_1 x_1, \lambda_2 x_2)$ , and the two coordinates never

interact. So the analysis from the last lesson applies to each one separately,

with its own curvature.

The lesson stops here

8 more paragraphs to go

You have read the opening. The rest of the argument, the problems that check whether it landed, and the lines worth keeping at the end all come with a plan.

The first lesson of every course in the library reads the whole way through, free, so you can see exactly what the rest of them are.

See the plan The contents

This is the reading half

Starting the course gives you your own copy of it. Every idea on every page has problems standing under it, marked with a reason rather than a tick, and any sentence you do not believe can be opened and argued with. None of that can happen on a page nobody owns.

The contents

One Ratio Decides How Long Training Takes

Two directions at once

8 more paragraphs to go

This is the reading half

Read alongside