The Simplest Model That Learns
Which Way to Turn the Knobs
Last timeMeasuring How Wrong It Is
The slope of the score with respect to each knob says which way is downhill, and stepping that way repeatedly is the method every model in use is fitted with.
Standing somewhere on the landscape
The score is a landscape over the knobs and fitting is finding its lowest
point. Suppose the knobs are set to a slope of two and an offset of a hundred
and twenty, which the last lesson scored at about nine hundred.
Nothing in that situation reveals where the lowest point is. What can be
discovered cheaply is which way is downhill, and that is enough, because a
walker who can always find downhill will reach the bottom of a bowl from
anywhere in it.
Finding downhill takes one idea: nudge a knob slightly and see whether the
score goes up or down.
The slope in one knob
Increase the slope knob by a thousandth and recompute the score. If the score
rose, the knob should go the other way. If it fell, continue in that
direction. Divide the change in score by the size of the nudge and the result
is a rate: how much the score changes per unit of knob.
That rate is the derivative, and it can be computed exactly rather than by
nudging. For the squared score and a straight line, the arithmetic is short
enough to do once and then never again.
- the error on example i, positive when the model guessed too high
- the input of example i
- the number of examples
The signs are worth reading rather than trusting. If the model is on average
guessing too high, the errors are positive, so both slopes are positive, so
both knobs should come down. That is exactly right: lowering either the offset
or the slope lowers the predictions. The formulas are saying something
obvious, which is the right state for a formula to be in the first time it
appears.
The lesson stops here
6 more paragraphs to go
You have read the opening. The rest of the argument, the problems that check whether it landed, and the lines worth keeping at the end all come with a plan.
The first lesson of every course in the library reads the whole way through, free, so you can see exactly what the rest of them are.
See the planThe contentsThis is the reading half
Starting the course gives you your own copy of it. Every idea on every page has problems standing under it, marked with a reason rather than a tick, and any sentence you do not believe can be opened and argued with. None of that can happen on a page nobody owns.
The contents