The Simplest Model That Learns
When It Learns the Noise
Last timePredicting Yes or No
A model with enough knobs will fit the accidents in its training data, and the cure is to hold the knobs back with a penalty and to measure on data the fitting never saw.
A model that is too good
Take ten data points that follow a gentle curve with a little measurement
error. Fit a straight line and it misses, because the relationship is curved.
Fit a model with ten knobs, using the input, its square, its cube and so on,
and it passes through every point exactly. The training score is zero.
It is also useless. Between the points it swings violently, and asked about
an input slightly beyond the data it returns something absurd. It has fitted
the measurement error, which is to say it has learned the accidents of these
particular ten observations as though they were the relationship.
This is the central failure of the whole field and it has a particular shape.
Add flexibility and the training score falls, always, without limit. The score
on data the model has not seen falls at first, because real structure is being
captured, and then rises, because accidents are being captured instead.
Holding data back
The gap in that picture is only visible because there were two scores. With
one, it is invisible.
So data must be held back: fit on part of it and measure on the rest. The held
back part has to be chosen before anything else happens, and then genuinely
not used. Not looked at while choosing which inputs to include, not used for
choosing the step size, not peeked at to see how things are going.
The reason for that severity is that any use at all is a form of learning from
it. A person who tries twenty settings and keeps whichever scored best on the
held back data has fitted that data by hand, and its score is now optimistic
in exactly the way the training score was. The standard arrangement has three
parts: one for fitting, one for choices made by hand, and one opened once at
the end.
The lesson stops here
6 more paragraphs to go
You have read the opening. The rest of the argument, the problems that check whether it landed, and the lines worth keeping at the end all come with a plan.
The first lesson of every course in the library reads the whole way through, free, so you can see exactly what the rest of them are.
See the planThe contentsThis is the reading half
Starting the course gives you your own copy of it. Every idea on every page has problems standing under it, marked with a reason rather than a tick, and any sentence you do not believe can be opened and argued with. None of that can happen on a page nobody owns.
The contents