From a Straight Line to Backpropagation
When the Derivative Equations Stop Being Solvable
Last timeSolving It Exactly, Once
Put a squashing function after the line and the two equations that gave w = 3 and b = 2 cannot be solved. What survives is the slope at the point you are standing on.
The twelve measurements have an exact best line, reached by writing two
equations and solving them. The slope is 3, the offset is 2, the
total squared error is 12, and nothing about that answer was approximate.
It is worth being precise about why that worked, because the next model breaks
exactly one thing about it. The score was
$$
and each measurement's residual, the measurement minus the prediction, was
, so the amount by which the prediction overshot was
. Either way round, the unknowns appeared multiplied by known
numbers and nothing else. Differentiating a sum of squares of such things
gives a sum of terms each linear in the unknowns, and twelve linear terms
collapse into one by adding up the numbers in front. That collapse is the whole
trick. It is what turned twelve measurements into the two equations
and , and two linear equations in two
unknowns are solved by elimination.
A model that can bend
A straight line through these points leaves a total squared error of 12, which
is small, but a line cannot bend at all. The smallest change that lets the model
bend is to push the line through a squashing function and then stretch and shift
the result.
- The old straight line, now squeezed into the interval from minus one to one. This is the only new thing in the model and it is the thing that will cost the closed form.
- An offset applied after the squashing, which the model needs because the squashed part can only reach from minus v to plus v.
- The input, scaled to run from one twelfth to one. Without this the squashing function is flat at every measurement and its derivative is zero to eleven decimal places.
Fix the four parameters at , , and to have
something to compute with. On the measurement at , where the recorded
value is , the scaled input is and the squashing
function is needed at . By its definition in terms of exponentials,
$$\tanh\left(\tfrac{1}{3}\right) = \frac{e^{2/3} - 1}{e^{2/3} + 1}
= \frac{1.9477340 - 1}{1.9477340 + 1} = \frac{0.9477340}{2.9477340}
= 0.32151274$$
so the prediction is .
The prediction is too high by , and the square of
that gap, which is this measurement's contribution to the total, is
.
Across all twelve measurements those contributions add to , against the
straight line's 12. The bent model has been handed four parameters and is
currently far worse than the two parameter line, because these four numbers
were chosen for convenient arithmetic rather than for fit. The question is how
to improve them.
The lesson stops here
4 more paragraphs to go
You have read the opening. The rest of the argument, the problems that check whether it landed, and the lines worth keeping at the end all come with a plan.
The first lesson of every course in the library reads the whole way through, free, so you can see exactly what the rest of them are.
See the planThe contentsThis is the reading half
Starting the course gives you your own copy of it. Every idea on every page has problems standing under it, marked with a reason rather than a tick, and any sentence you do not believe can be opened and argued with. None of that can happen on a page nobody owns.
The contents