Basis Lab

From a Straight Line to Backpropagation

Running It, and Reading the Curve It Produces

Last timePaying for the Shared Part Once

The finished method on the original twelve points, with an error curve whose floor is twelve rather than zero, and three misbehaving curves with the number to change for each.

Everything the method needs is in hand: a number that says how wrong a set of

parameters is, a way of getting its derivative with respect to every parameter

for the price of a few evaluations, and a bound on how far a step may go. What

is left is to run it on the twelve measurements it started with, $y = 6, 7, 10,

15, 18, 19, 22, 27, 30, 31, 34, 39atatx = 1$ through 12, and to read the curve

it produces, because that curve is the only thing anybody watches. Nothing has

been held back for a test: the run sees all twelve.

Which model the run uses, and why that one

The run below fits a straight line to the centred inputs rather than the bent

two unit network, for one reason: the straight line's error curve can be written

as a formula, so every number the run produces can be checked against it and

every diagnosis can be seen to be right rather than merely plausible. The bent

network has no such formula, and the last section says what carries over.

So write zi=xi−6.5z_i = x_i - 6.5 for the centred inputs, as the bound on the step

size required, and fit y^=w′z+b′\hat{y} = w' z + b'. Centring makes ∑zi=0\sum z_i = 0, and

with ∑zi2=143\sum z_i^2 = 143 and ∑ziyi=429\sum z_i y_i = 429 the two equations separate at

once:

$w′=429143=3,b′=yˉ=25812=21.5w' = \frac{429}{143} = 3, \qquad b' = \bar{y} = \frac{258}{12} = 21.5$

Undoing the centring gives

b=21.5−3(6.5)=21.5−19.5=2b = 21.5 - 3(6.5) = 21.5 - 19.5 = 2, so this is the line w=3w = 3, b=2b = 2 the

exact solution produced earlier. Taking a residual as the measurement minus the

prediction, ri=yi−y^ir_i = y_i - \hat{y}_i, the twelve residuals are +1,−1,−1,+1+1, -1, -1, +1

repeating, and the total squared error is twelve.

The error surface, exactly

Let d=w′−3d = w' - 3 and c=b′−21.5c = b' - 21.5 measure how far the parameters are from the

answer, the same two deviations the step size bound was written in, and write

rir_i for the residuals of the exact line. Moving the slope by dd and the

intercept by cc raises each prediction by dzi+cd z_i + c, so the measurement minus

the prediction becomes ri−dzi−cr_i - d z_i - c, and squaring and summing gives

$E=∑ri2−2d∑rizi−2c∑ri+d2∑zi2+2dc∑zi+12c2E = \sum r_i^2 - 2d \sum r_i z_i - 2c \sum r_i + d^2 \sum z_i^2 + 2dc \sum z_i + 12 c^2$

Three of those six sums vanish. ∑ri=0\sum r_i = 0 and ∑rizi=0\sum r_i z_i = 0 are the two

normal equations themselves, and ∑zi=0\sum z_i = 0 is the centring. What is left is

exact, with no approximation anywhere:

The lesson stops here

10 more paragraphs to go

You have read the opening. The rest of the argument, the problems that check whether it landed, and the lines worth keeping at the end all come with a plan.

The first lesson of every course in the library reads the whole way through, free, so you can see exactly what the rest of them are.

See the planThe contents

This is the reading half

Starting the course gives you your own copy of it. Every idea on every page has problems standing under it, marked with a reason rather than a tick, and any sentence you do not believe can be opened and argued with. None of that can happen on a page nobody owns.

The contents

The rest of this course

  1. 01Measuring How Wrong a Straight Line Is, on Twelve Points
  2. 02The Two Equations That Pin Down the Best Lineopening only
  3. 03When the Derivative Equations Stop Being Solvableopening only
  4. 04The Largest Step This Data Allowsopening only
  5. 05Putting a Bend in the Model, and Differentiating Through Itopening only
  6. 06Every Derivative of a Two Unit Network, by Handopening only
  7. 07One Sweep Back Instead of Seven Chainsopening only
  8. 08Running It, and Reading the Curve It Producesyou are here

Read alongside