Three Ways a Run Breaks, and What Each One Looks Like
Last timeReading a Run While It Happens
A loss spike, a divergence and a dead plateau look alike for about ten steps and need completely different responses. Here is how to tell them apart.
Three shapes
Runs break in a small number of ways, and the shapes are distinct enough to name.
A spike is a sudden jump in the loss followed by a return to roughly where
it was. It is not on that chart because it is a single point: the loss goes from
2.2 to 7 for one step and is back at 2.4 a hundred steps later.
A divergence is a jump that does not come back. The loss climbs, or goes to
infinity, or becomes the special value that means "not a number" and stays there
for the rest of the run.
A dead plateau is a loss that is flat from the beginning, usually at a
suspiciously specific value, and never moves.
The responses are different enough that getting the diagnosis wrong is
expensive. A spike treated as a divergence throws away a working run. A
divergence treated as a spike wastes however many hours it takes to notice.
A spike, and what it costs to undo
A spike is almost always one batch. Something in it produces an unusually large
gradient, the step lands somewhere bad, and the next few hundred steps are the
model finding its way back.
Most spikes resolve themselves, and the right response to most of them is
nothing. The decision worth making in advance is at what point you stop waiting,
and the answer comes out of your checkpointing interval, because rolling back
costs exactly the work since the last checkpoint.
The lesson stops here
6 more paragraphs to go
You have read the opening. The rest of the argument, the problems that check whether it landed, and the lines worth keeping at the end all come with a plan.
The first lesson of every course in the library reads the whole way through, free, so you can see exactly what the rest of them are.
See the planThe contentsThis is the reading half
Starting the course gives you your own copy of it. Every idea on every page has problems standing under it, marked with a reason rather than a tick, and any sentence you do not believe can be opened and argued with. None of that can happen on a page nobody owns.
The contents