Charging For The Wrong Answers
Last timeToo Simple And Too Complicated
Every way of fighting overfitting is the same move, which is making some settings expensive so that the data has to insist before they are reached.
The previous lesson left a clear instruction. If a model is too sensitive to its
training sample, reduce the freedom the fitting procedure has rather than shrinking
the family it chooses from. Every technique in this lesson does that, and they all
do it the same way: by making some answers more expensive than others, so that the
data has to insist before an expensive one is adopted.
A price rather than a prohibition
The direct version adds a term to the quantity being minimised. Instead of
minimising the training error alone, the procedure minimises the training error plus
a charge that grows with the size of the settings.
- the adjustable quantities of the model
- the cost of being wrong on one example
- the charge, which grows as the settings get larger
- the strength of the charge, chosen by the person not by the fit
This is why a penalty reduces sensitivity without introducing rigidity in the sense
of the previous lesson. The family is unchanged and still contains the truth. What
has shrunk is the set of settings the run will actually arrive at, which is the
effective capacity from the capacity lesson, reached by charging rather than by
redesigning.
The lesson stops here
6 more paragraphs to go
You have read the opening. The rest of the argument, the problems that check whether it landed, and the lines worth keeping at the end all come with a plan.
The first lesson of every course in the library reads the whole way through, free, so you can see exactly what the rest of them are.
See the planThe contentsThis is the reading half
Starting the course gives you your own copy of it. Every idea on every page has problems standing under it, marked with a reason rather than a tick, and any sentence you do not believe can be opened and argued with. None of that can happen on a page nobody owns.
The contents