Putting a Bend in the Model, and Differentiating Through It

The twelve measurements have already been fitted exactly. Two derivatives set

to zero gave two equations, the two equations gave $w = 3$ and $b = 2$, and at

that answer the total squared error is 12, a mean of 1 per measurement. Nothing

in it was approximate and nothing in it was iterative.

A line pays for that exactness by having one slope and using it everywhere.

Whatever it does at the left-hand end of the data it does again at the

right-hand end. Most measured quantities do not behave like that: a response

that grows and then levels off, a dose that stops helping, a queue that only

gets slower until the machine saturates and then goes vertical. A model that

cannot flatten will fit the middle and be wrong at both ends, and no amount of

solving rescues it, because the failure is in the shape of the model rather

than in the values of its parameters.

So the model gets a bend in it. The bill arrives immediately, because the

closed form goes and does not come back, and the only thing left is the slope

of the error where you happen to be standing. This page computes that slope,

slowly, on one measurement.

The bent model itself is already established: the input scaled as $u = x/12$,

the old straight line rebuilt on the scaled input as $a = wu + b$, the squashed

value $t = \tanh a$ lying between minus one and one, and $p = vt + c$ putting

that back on the scale of measurements running from 6 to 39. Two things about

it are settled and are not argued again here: that $w$ sits inside the

squashing function, so the twelve terms of a derivative cannot be collected and

there is no formula for the answer, and that dividing the input by twelve is

what keeps the arguments off the flat part of the curve. The one fact needed now

is the shape. Four steps separate the input from the prediction, each with one

input and one output, and that is what makes the derivative a product.