The Largest Step This Data Allows

Last timeWhen Exact Runs Out

The twelve points fix a curvature, the curvature fixes a largest safe step of 0.001516, and centring the inputs raises that to 0.006993 without changing a single measurement.

Stepping against the slope leaves a method with a hole in it. Evaluate it, step

against it, repeat, and one step of size 0.01 on a single measurement dropped

the squared error by 1.804759 where the slope had promised 1.824894. The

promise grows in proportion to the step and the shortfall grows as its square,

so a large enough step stops helping and then starts hurting. Nothing in the

gradient says where that happens.

The straight line on the twelve points answers the question exactly, which is

why the question is asked here rather than on the bent model. The score is

$ $E(w, b) = 650w^2 + 156wb + 12b^2 - 4212w - 516b + 6846$ $

and written around its minimum that is

$ $E(3 + d, 2 + c) = 12 + 650d^2 + 156dc + 12c^2$ $

where $d$ is how far the slope sits from 3 and $c$ is how far the offset sits

from 2. Everything in this lesson is about the three numbers 650, 156 and 12 in

that expression.

Curvature is already written down

Curvature is the second derivative, and a quadratic wears its second

derivatives on its face.

FIG 1The curvature of the twelve point score

: the matrix of second derivatives of the total squared error, one entry for each pair of parameters
: the sum of x squared over the twelve measurements
: the sum of x over the twelve measurements
: the number of measurements

No differentiation was needed. Twice the coefficient of d squared is 1300, twice the coefficient of c squared is 24, and the coefficient of the cross term d times c is the mixed second derivative, 156. Unlike the gradient, these four numbers do not depend on where the parameters currently are, which is what makes a single step ceiling possible for the whole run. The 1300 against the 24 is the asymmetry that one step on one measurement pointed at and did not use.

Those numbers already say something uncomfortable. Moving the slope by one unit

costs 650 in the score and moving the offset by one unit costs 12. The slope

direction is 54 times sharper than the offset direction, and not because the

measurements are strange: the sum of $x$ squared is 650 only because $x$ runs

from 1 to 12, so every measurement pulls the slope hard. The cross term says

worse. A step that changes the slope alone also changes the score through

$156dc$ , so slope and offset are not separate directions and tuning a step for

one of them is tuning it for a mixture.

The lesson stops here

9 more paragraphs to go

You have read the opening. The rest of the argument, the problems that check whether it landed, and the lines worth keeping at the end all come with a plan.

The first lesson of every course in the library reads the whole way through, free, so you can see exactly what the rest of them are.

See the plan The contents

This is the reading half

Starting the course gives you your own copy of it. Every idea on every page has problems standing under it, marked with a reason rather than a tick, and any sentence you do not believe can be opened and argued with. None of that can happen on a page nobody owns.

The contents

The Largest Step This Data Allows

Curvature is already written down

9 more paragraphs to go

This is the reading half

Read alongside