Carry one small set of measurements from fitting a straight line by hand all the way to training a two layer network, so that every piece of modern training arrives as the repair of something that broke on the previous page

From a Straight Line to Backpropagation

One dataset, twelve points, carried from a line fitted by hand to a trained two layer network, with every step forced by the failure of the step before.

8 lessons, written and corrected before you arrived. Reading them here needs no account. The first reads the whole way through; the others open and then stop, because a page nobody owns cannot tell who is reading it. Starting the course gives you your own copy, where every idea has problems standing under it and you can ask about any sentence.

Start reading

01Measuring How Wrong a Straight Line Is, on Twelve PointsTwelve measurements, two candidate lines and the arithmetic that scores them, with the totals 12 and 24 worked out point by point and the choice of squaring argued rather than assumed.
02The Two Equations That Pin Down the Best Lineopening onlySetting both partial derivatives of the squared total to zero gives two linear equations in the slope and the intercept, 78w + 12b = 258 and 650w + 78b = 2106, solved here by hand to w = 3 and b = 2.
03When the Derivative Equations Stop Being Solvableopening onlyPut a squashing function after the line and the two equations that gave w = 3 and b = 2 cannot be solved. What survives is the slope at the point you are standing on.
04The Largest Step This Data Allowsopening onlyThe twelve points fix a curvature, the curvature fixes a largest safe step of 0.001516, and centring the inputs raises that to 0.006993 without changing a single measurement.
05Putting a Bend in the Model, and Differentiating Through Itopening onlyA squashing function after the line makes the model a composition, so the derivative becomes a product of four local slopes, worked here by hand on one measurement.
06Every Derivative of a Two Unit Network, by Handopening onlyTwo squashing units give seven parameters and seven derivatives, all computed here on one measurement, and the same three numbers turn up in six of them.
07One Sweep Back Instead of Seven Chainsopening onlyThe seven derivatives of a two unit network are built out of the same two numbers, and computing those once and sweeping backwards costs about three forward passes.
08Running It, and Reading the Curve It Producesopening onlyThe finished method on the original twelve points, with an error curve whose floor is twelve rather than zero, and three misbehaving curves with the number to change for each.