Answers · How models work

Why does training loss keep falling while validation loss starts rising?

In shortTraining loss keeps falling because the model is memorising the batches it is shown; held-out loss rises when that memorisation stops helping on data it has not seen.

Training loss keeps falling because the model is getting better at the particular batches it is being shown. Held-out loss rises when that improvement stops transferring to data the model has not seen: the model is fitting noise and specifics rather than the task. The two numbers answer different questions, and only the second one tells you whether the deliverable has improved. A falling training loss is a statement about the batches already seen, not about the next batch from the same source. It is the easiest number to log and the easiest to misread. A model can improve its training loss every step while getting worse on everything it will be judged on.

The gap, worked out

Write the gap as $g = L_{\text{held}} - L_{\text{train}}$ . Take a run at step 40000 with training loss 2.21 and held-out loss 2.33. The gap is $2.33 - 2.21 = 0.12$ . At step 60000 the training loss has fallen to 2.04, a fall of $2.21 - 2.04 = 0.17$ , while held-out loss has risen to 2.41, a rise of $2.41 - 2.33 = 0.08$ . The gap is now $2.41 - 2.04 = 0.37$ . Training is still improving on its own batches. The model is getting worse on everything else.

g = L_{\text{held}} - L_{\text{train}}

FIG 1

step	L_train	L_held	g
40000	2.21	2.33	0.12
60000	2.04	2.41	0.37

Why the natural guess fails

The natural guess is that any fall in loss means the model is improving. That would be true only if the batches were a fair sample of everything the model will meet. They are not. A model with enough capacity can reduce training loss towards zero on a finite training set by memorising the examples. That curve looks exactly like a healthy run, and it is why training loss alone cannot distinguish learning from memorising. The held-out curve is the check: it uses data the model is not being trained on, so memorisation of the training set does not help there. When the held-out curve turns upward, further steps are making the model better at the training examples and worse at the task.

The held-out curve is not sampled every step because evaluation costs time. It is measured every few hundred or few thousand steps. That makes it a coarse view. A single upward reading can be noise, so the signal is a sustained turn across several evaluations, not one point. The moment the held-out curve stops falling is the moment further training stops being worth anything, whatever the training curve is doing. If it then rises, every further step is making the deliverable worse.

FIG 2Two curves that start together

trainingheld out

The training curve keeps falling. The held-out curve falls at first, then turns upward once the linear term overtakes the decaying term.

What the gap is telling you

A small stable gap is healthy. A small gap is normal even in a good run because training batches are slightly easier than held-out data. What matters is not the size of $g$ at one evaluation but whether $g$ is growing. A gap that grows steadily while both curves still fall is a warning: the model is beginning to fit specifics of the training set. A gap that grows because $L_{\text{held}}$ has turned upward is not a warning, it is the emergency. The response is to stop, because the checkpoint you keep is the one that will be used, and later checkpoints are worse on the held-out data.

The early warning that is not the loss

The gradient norm often moves before either loss curve does. In a run heading for divergence the gradient norm can climb over tens of steps while the training loss is still flat or falling. By the time the loss reacts, the weights have already taken several large steps. Log the gradient norm before clipping and whether the clip fired. A clip that fires on one step in a thousand is insurance. A clip firing on half the steps means the configured and actual step sizes have come apart. This does not explain a held-out curve that has already turned, but it can give you the tens of steps before a break.

Where this reading stops

The reading assumes the held-out set is genuinely unseen. If any of it has leaked into training, $L_{\text{held}}$ will fall for the wrong reason and the gap will look healthy when it is not. If training and held-out data come from different distributions, a rising held-out loss can mean the distribution changed, not that the model started memorising. If the held-out evaluation is too coarse or too small, one upward point can be noise. Check the turn across several evaluations and against a fixed reference before treating it as the end of the run. Throughput belongs on the same dashboard for a different reason: a run that has halved in speed still shows a perfect loss against step number and doubles the bill against wall-clock time.

Worth keeping

Training loss falling is not evidence that held-out loss is falling.
The gap is held-out loss minus training loss, and a growing gap is the warning; a gap growing because held-out loss has turned upward is the emergency.
When held-out loss has turned upward, further training is making the deliverable worse.

Where this goes next

This question is one step of Training a Model From Scratch, a course that works through its subject from the beginning. The page that teaches this part of it is Reading a Run While It Happens, and it reads the whole way through whether or not you have an account.

What an account changes is the practice: problems standing under every idea, marked with a reason rather than a tick, in a copy of the course that is yours.

Read that page The whole course