From a Straight Line to Backpropagation
Every Derivative of a Two Unit Network, by Hand
Last timeBending the Line
Two squashing units give seven parameters and seven derivatives, all computed here on one measurement, and the same three numbers turn up in six of them.
One squashing unit can bend, but it can only bend one way. The hyperbolic
tangent rises everywhere, and multiplying it by an output weight can turn it
upside down but cannot make it rise and then fall. So a model built from a
single unit has exactly one shape available to it, in two mirror images: up and
flattening, or down and flattening. A response that climbs, peaks and falls
away is not in the family, at any setting of the four parameters.
Two units are. Add a second squashing unit and sum the two, and if their output
weights have opposite signs the sum can turn over. That is a genuinely new
repertoire, and the price is three more parameters and, as it turns out, seven
derivatives to compute instead of four. This page computes all seven, by hand,
on one measurement, and lays the arithmetic out so that what is being
recomputed is impossible to miss.
Two units, seven parameters
The input is scaled as before, . Each unit has its own input weight
and input offset, forms its own pre-activation, and squashes it. The prediction
is a weighted sum of the two squashed values plus a single offset.
- the scaled input, between one twelfth and one across the twelve measurements
- which unit, 1 or 2, so every symbol carrying it comes in a pair
- unit j's input weight, setting how fast it traverses its squashing curve
- unit j's input offset, setting which part of the curve it sits on
- unit j's pre-activation, a straight line in the scaled input
- unit j's squashed value, between minus one and one
- unit j's output weight, which can be negative, and that is what lets the sum turn over
- the single output offset, shared by both units, which is why the second unit costs three parameters rather than four
The starting values are , , for the first unit,
, , for the second, and . The second
unit's negative input weight means it falls where the first one rises, which is
the configuration worth starting from, since two units pointing the same way
are nearly one unit with a larger output weight.
At these values the model is not a serious attempt at the twelve
measurements. It reads 19.7789 at where the measurement is 15, and the
exact line reads 14 there, so it is nearly five too high at a point the line
gets almost right. That is deliberate. The parameters were chosen so that
neither unit is saturated, both local slopes are of a usable size, and the
seven derivatives are all large enough to be read off and checked by hand.
Starting from a good fit would make every derivative small and every arithmetic
slip invisible.
The lesson stops here
6 more paragraphs to go
You have read the opening. The rest of the argument, the problems that check whether it landed, and the lines worth keeping at the end all come with a plan.
The first lesson of every course in the library reads the whole way through, free, so you can see exactly what the rest of them are.
See the planThe contentsThis is the reading half
Starting the course gives you your own copy of it. Every idea on every page has problems standing under it, marked with a reason rather than a tick, and any sentence you do not believe can be opened and argued with. None of that can happen on a page nobody owns.
The contents