What A Training Curve Is Telling You
Last timeThe Same Trouble Backwards
Everything in this course arrives in practice as one line on a chart going the wrong way, so the last skill is reading that line and knowing which of the remedies to reach for.
Seven lessons of mechanism arrive in practice as one line on a chart that is not
doing what it should. This lesson is about that line: what it can tell you, what
it cannot, and what to measure next.
Start with the honest limitation. The loss curve is the cheapest instrument you
have, because it is recorded anyway, and it is also the least specific. A curve
that falls and then flattens at a disappointing value is consistent with a
vanishing gradient, a step size far too small, a model too small for the task, and
a mistake in how the data is labelled. Those four have nothing in common and the
curve does not distinguish them.
What a spike is and what is usually done about it
A spike is a sudden large rise in a run that was descending normally. An enormous
gradient arrived, the step moved the parameters somewhere much worse, and the run
has to climb back out.
The published accounts of large training runs are unusually consistent here. One
reports roughly twenty spikes in a single run and describes the response: go back
to a saved state from a few hundred steps before the spike, skip the batches that
were being processed when it happened, and continue. Interestingly, the same
batches usually pass without incident when the run reaches them from a slightly
different state, which says the batch was not corrupt. It was an unlucky
interaction between a particular batch and a particular set of parameters.
| step | step | loss | overall gradient length | what was done | what happened |
|---|---|---|---|---|---|
| 1 | 21400 | 2.61 | 0.42 | nothing, a normal step | The run is descending at the expected rate and the gradient length is typical for this stage. |
| 2 | 21418 | 2.64 | 18 | clipped to the threshold | A large gradient arrives. Clipping bounds the step, and on most occasions this is the end of the story. |
| 3 | 21419 | 3.95 | 6.1 | nothing | The loss has risen sharply anyway. The clipped step was still enough to move the parameters into a much worse region. |
| 4 | 21460 | 3.4 | 2.2 | nothing | The run is climbing back out on its own, slowly, and will take thousands of steps to recover the ground it lost. |
| 5 | 21400 | 2.61 | 0.42 | restarted from the saved state, batches | The intervention: reload the state from before the spike, skip the batches around step 21418, and continue from there. |
Three numbers worth recording
The loss alone leaves you guessing. Three further measurements, all cheap, turn a
vague curve into a named fault.
The length of the gradient at each step, which shows an explosion directly and also
shows how often clipping is firing. If it fires on most steps the threshold is
acting as a cap on the step size rather than as a safety measure.
The size of each update relative to the size of the parameter it changes. This is
more informative than the gradient on its own, because the optimiser rescales
gradients and a gradient of a given size says little about how far anything
actually moved. It is also the quantity that has been found to predict instability
at larger scale when measured in small runs.
The spread of the values at several depths, which shows the forward problem from
the second lesson and, more usefully, shows where in the stack it starts.
The lesson stops here
2 more paragraphs to go
You have read the opening. The rest of the argument, the problems that check whether it landed, and the lines worth keeping at the end all come with a plan.
The first lesson of every course in the library reads the whole way through, free, so you can see exactly what the rest of them are.
See the planThe contentsThis is the reading half
Starting the course gives you your own copy of it. Every idea on every page has problems standing under it, marked with a reason rather than a tick, and any sentence you do not believe can be opened and argued with. None of that can happen on a page nobody owns.
The contents