Answers · How models work
Why did my loss suddenly spike and can I recover without restarting?
In shortYour loss spike is probably one bad batch, and you can usually recover by waiting or rolling back to a recent checkpoint instead of restarting the whole run.
Your run most likely hit one bad batch that produced an unusually large gradient and one bad step; if the loss is coming back within a few hundred steps, that is a spike, not a divergence. You can recover without restarting the whole run by waiting for the return, or by rolling back to the last checkpoint and resuming with the batches in that window skipped; the cost is the work since that checkpoint, not the whole training run.
The shape of a spike
A spike has a fingerprint: the loss jumps on one step and the gradient norm jumps with it. At step 41,201 the loss goes from 2.21 to 7.84, a jump of 5.63, while the gradient norm goes from 0.33 to 12.60. By step 41,260, which is 59 steps later, the loss is 3.10 and the gradient norm is 0.51; by step 41,500, which is 299 steps after the spike, the loss is 2.29, only 0.08 above where it was before. The model is finding its way back. The correct response to most spikes is to do nothing and keep the run.
The expensive guess
The tempting response is to restart from scratch because the spike looks like corruption. That fails the arithmetic: if your last checkpoint is at step 41,000 and the spike is at 41,201, rolling back discards 201 steps; restarting discards 41,201 steps. If checkpoints are every 2,000 steps, the last one before the spike is at 40,000, and rolling back discards 1,201 steps, which is still small next to the whole run. A spike treated as a divergence throws away a working run, and a divergence treated as a spike wastes the hours spent waiting for a recovery that cannot happen.
When waiting stops being right
The decision to make in advance is when to stop waiting, and that threshold comes from your checkpoint interval. If the loss is still at 7 by step 41,500, 299 steps after the spike, you are no longer looking at a spike. Roll back to the last checkpoint before the bad batch and resume with that window of batches skipped. This is not a full restart: it is a partial rewind whose cost is exactly the work since the checkpoint. Frequent checkpoints are cheap insurance because the interval is the maximum amount of work one bad batch can destroy.
When it is divergence
A divergence is a different mechanism: large weights produce large activations, large activations produce large gradients, large gradients produce a large step, and the large step produces larger weights. Once that loop closes, waiting does not help. If the loss climbs to infinity or becomes NaN and stays there, the current run has no information left, and continuing only writes dead checkpoints. Recover by going back to a checkpoint from before the divergence and changing the cause: clip the gradient norm, warm the learning rate up, fix normalisation, or keep certain sums in a wider number format than the weights.
A dead plateau is the third shape and it is not a spike: the loss is flat from the beginning, usually at the loss of a uniform guess, which is the natural logarithm of the vocabulary size, . It does not jump and return, and the gradients are not reaching the weights at all. The causes are mechanical: a learning rate of zero, a frozen parameter group, gradients not being propagated, or labels shuffled independently of the inputs. That is a disconnected run, not a slow one.
Check the data first
Before touching the optimiser, print the batch at the step where it broke, decoded back into text. A file with one enormous unbroken line, a block of identical tokens, or binary that got through the filter will produce exactly the kind of enormous gradient that makes a spike, and it is minutes of work to see. Then check the scale of things: the gradient norm, the size of the weights, and whether the loss at step zero matched the loss of a uniform guess. Only after those should you look at the learning rate actually in force, whether clipping is firing, and whether gradients were cleared between steps. The optimiser feels like the part doing the learning, so it is where people look first, and it is the least likely culprit and the most expensive thing to experiment with.
Worth keeping
- A spike is one bad step whose loss returns within a few hundred steps; a divergence does not return, and a run at NaN has no information left.
- Rolling back to a checkpoint costs exactly the work since that checkpoint, so frequent checkpoints bound the damage one bad batch can do.
- Check the decoded batch before changing the learning rate, because broken input is more often the cause than the optimiser.
Where this goes next
This question is one step of Training a Model From Scratch, a course that works through its subject from the beginning. The page that teaches this part of it is Spikes, Plateaus and Dead Runs, and it reads the whole way through whether or not you have an account.
What an account changes is the practice: problems standing under every idea, marked with a reason rather than a tick, in a copy of the course that is yours.
Read that pageThe whole courseOther questions people ask about this
- Why do we need a forward pass before we can backpropagate?The gradient is built from local derivatives evaluated at the values the forward pass computed, so the loss alone cannot supply it.
- Why does a bigger batch make the gradient less noisy?Each step's gradient is an average, and an average's error shrinks with the square root of the number of examples, so four times the batch only halves the noise.
- Why train on a loss function instead of just using accuracy?Because accuracy is a pass or fail count that is flat almost everywhere, so it gives the optimiser no slope to follow, while a loss changes smoothly and tells every weight which way to move.
- Why does training loss keep falling while validation loss starts rising?Training loss keeps falling because the model is memorising the batches it is shown; held-out loss rises when that memorisation stops helping on data it has not seen.
- Why does the model get stuck repeating the same sentence over and over?The model is not stuck in a memory loop: once it has written a phrase, that phrase is in its context and makes writing it again more likely, so a decoder that always picks the top word cannot leave.
- Why does diffusion add Gaussian noise instead of any random corruption?Diffusion adds Gaussian noise because Gaussian errors combine exactly: each step keeps the spread fixed and gives a closed-form noisy picture at any later step, which makes the reverse step learnable.