Past The Point Where It Should Break
Last timeWhat Another Thousand Examples Buy
Error rises to a peak where a model has just enough capacity to fit the training data exactly, and then falls again as capacity grows past it.
The classical curve says that past a certain capacity things get worse and keep
getting worse. Practice says otherwise, loudly. Models with far more adjustable
quantities than training examples, fitted until their training error is exactly zero,
turn out to work well, and they are the models everything now runs on. Something in
the picture from the tradeoff lesson is incomplete, and the missing piece was
described carefully only in 2019.
What the measurement found
Take any family you can grow smoothly, hold the dataset fixed, and measure held-out
error at many capacities rather than at a handful. For small capacities you see the
familiar fall. Then you see a rise, exactly as predicted. The rise does not continue.
It reaches a sharp peak and then falls again, often ending below the lowest point of
the first descent.
The peak is the informative part. It does not sit at some arbitrary large capacity.
It sits at the point where the number of free quantities roughly matches the number
of training examples, which is the point at which the model can first pass exactly
through every training example and has nothing left over.
The lesson stops here
5 more paragraphs to go
You have read the opening. The rest of the argument, the problems that check whether it landed, and the lines worth keeping at the end all come with a plan.
The first lesson of every course in the library reads the whole way through, free, so you can see exactly what the rest of them are.
See the planThe contentsThis is the reading half
Starting the course gives you your own copy of it. Every idea on every page has problems standing under it, marked with a reason rather than a tick, and any sentence you do not believe can be opened and argued with. None of that can happen on a page nobody owns.
The contents