The Four Numbers Worth Watching
Last timeHow Much to Look at First
A run reports hundreds of numbers and four of them matter: training loss, held-out loss, gradient size and throughput. Each fails in its own way.
Training loss is the number everyone watches
A training run prints one number constantly, and that number is the loss on the
batch it just saw. It is the easiest thing to log, it is available every step,
and it is the one people put on the screen.
It cannot answer the question anybody actually has. A falling training loss
establishes that the model is getting better at the examples it is being shown.
It does not establish that it is getting better at anything else, and there is a
specific way for those two to come apart: a model with enough capacity can
reduce the loss on a finite training set to nearly zero by memorising it, and
that curve looks exactly like a run going beautifully.
The fix is to measure the loss on data the model is not being trained on, every
so often, and put both curves on the same chart.
The moment the held-out curve stops falling is the moment further training stops
being worth anything, whatever the training curve is doing. That it eventually
turns upward is the clearest possible signal: the model is now getting worse at
the task while getting better at the examples.
The gap, and what to do about it
The distance between the two curves has a name and a straightforward reading.
The lesson stops here
5 more paragraphs to go
You have read the opening. The rest of the argument, the problems that check whether it landed, and the lines worth keeping at the end all come with a plan.
The first lesson of every course in the library reads the whole way through, free, so you can see exactly what the rest of them are.
See the planThe contentsThis is the reading half
Starting the course gives you your own copy of it. Every idea on every page has problems standing under it, marked with a reason rather than a tick, and any sentence you do not believe can be opened and argued with. None of that can happen on a page nobody owns.
The contents