Basis Lab

Answers · The mathematics underneath

Why does test error go down and then back up as I make the model more complicated?

In shortA more flexible model first removes its systematic miss, then starts fitting the noise in its particular training sample, so test error falls and then rises.

Test error falls and then rises because two avoidable errors move in opposite directions as capacity grows. The miss caused by a family too rigid to express the truth shrinks; the instability caused by a fit flexible enough to follow the particular sample's noise grows. Once the second movement is larger than the first, the total turns upward. Capacity here means the number of free quantities in the fitted family.

The two movements

Write kk for capacity, measured in free quantities. The expected error on a new example splits into three parts: a systematic miss of the average fit, the wander of an individual fit around that average, and the noise in the data. The first is large when the family cannot express the truth. The second is large when a fit has enough freedom to follow its own training sample, quirks included. The third is a floor.

E[(y−f^(x))2]=(fˉ(x)−f(x))2+E[(f^(x)−fˉ(x))2]+σ2E\left[(y-\hat{f}(x))^2\right] = \left(\bar{f}(x)-f(x)\right)^2 + E\left[(\hat{f}(x)-\bar{f}(x))^2\right] + \sigma^2

The average fˉ(x)\bar{f}(x) is taken over many training samples of the same size drawn from the same source, with the model refitted each time. The first term is the squared gap between that average and the truth. It does not shrink when you collect more data, because a rigid family is wrong in the same direction every time. The second term is the mean squared distance from an individual fit to that average. It does shrink with more data, because each fit is pinned down more firmly. The third term is the noise, and no model can remove it.

For a concrete illustration, let the rigidity part be 20/k20/k and the sensitivity part be 0.5k0.5k. Then the total is their sum. At k=2k=2 the rigidity part is 20/2=1020/2=10 and the sensitivity part is 0.5×2=10.5\times2=1, giving 1111. At k=6k=6 they are 20/6=3.3320/6=3.33 and 33, giving 6.336.33. At k=20k=20 they are 11 and 1010, giving 1111. The sum bottoms out where the two terms are equal in size in this illustration, near k=6.32k=6.32.

T(k)=20k+0.5k,T′(k)=−20k2+0.5T(k)=\frac{20}{k}+0.5k,\qquad T'(k)=-\frac{20}{k^2}+0.5
FIG 1
capacity rigidity sensitivity total
210.001.0011.00
45.002.007.00
63.333.006.33
72.863.506.36
102.005.007.00
141.437.008.43
201.0010.0011.00

The numbers are illustrative rather than fitted to a real dataset, but the shape is the point. From k=2k=2 to k=6k=6 the rigidity part falls by 10−3.33=6.6710-3.33=6.67 while the sensitivity part rises by 3.00−1.00=2.003.00-1.00=2.00; the fall wins. From k=6k=6 to k=20k=20 rigidity falls by only 3.33−1.00=2.333.33-1.00=2.33 while sensitivity rises by 10.00−3.00=7.0010.00-3.00=7.00; the rise wins. The total therefore has a minimum in between.

The rising side is not extra epochs

The natural guess is that the right-hand side is caused by training too long. At a fixed capacity, training longer usually lowers both training error and held-out error until the fit settles; if held-out error turns up while training error keeps falling, that is an early-stopping problem, and stopping earlier can help. The right arm of the capacity curve is different. It is present after each model has been trained to convergence, and it comes from the fit having enough free quantities to reproduce the particular noise in its sample. Retrain a high-capacity model on a different sample of the same size and its prediction for a fixed new point moves visibly. Retrain a low-capacity model and it barely moves. That instability, together with near-zero training error and a large gap, identifies sensitivity rather than excessive optimisation.

The minimum is not a fixed property of the model family. More training examples pin down each individual fit more firmly, so the sensitivity part shrinks at every capacity; the whole curve moves down and its minimum moves to the right. A model that is too flexible for 100 examples can be the right size for 10000. A penalty on size changes effective capacity in the same direction. The simple U also stops being the whole story for some large models, which can have zero training error and stable held-out error; for those, the classical picture is not a reliable guide, and the measured held-out error is.

Read training error and held-out error together. If both are high, you are on the left and more capacity, better inputs or a different family can help; more data will not remove a systematic miss. If training error is low and held-out error is much higher, you are on the right and more data, a penalty, early stopping or averaging several fits can help; adding freedom will not. If both are low and close, you are at the noise floor.

Worth keeping

  • Test error can rise as capacity grows even while training error is still falling or has reached zero.
  • The rising side is sensitivity to the particular training sample, not simply too many training epochs.
  • More training examples move the best capacity to the right and down.

Where this goes next

This question is one step of Why Learning Works At All, a course that works through its subject from the beginning. The page that teaches this part of it is Too Simple And Too Complicated, and it reads the whole way through whether or not you have an account.

What an account changes is the practice: problems standing under every idea, marked with a reason rather than a tick, in a copy of the course that is yours.

Read that pageThe whole course

Other questions people ask about this

  1. Why does my test set stop being a fair test if I keep tuning on it?Because every time you use the test set to choose something, information from it shapes the model, so it stops being untouched and its score becomes optimistic.
  2. Why can a model get 100% on the training set and still be useless?A perfect training score only shows the model can reproduce the examples it was fitted on; it says nothing about new examples, where it may be no better than guessing.
  3. Why does adding a penalty on the weights stop overfitting?Adding a penalty makes large weights costly, so the fit keeps them only when the training data saves enough error to pay for them, which limits the settings the model can reach.
  4. What is the difference between variance and standard deviation?They measure the same spread: variance is the average squared distance from the mean, standard deviation is its square root, so you report the standard deviation and calculate with the variance.
  5. How do you make a vague claim precise enough to tell whether it's true?Fix what is being talked about, what is asserted, which cases are covered and the cut-off, then name an observation that would prove it false.
  6. How do you find the parabola that best fits a function near a point?Match the function's height, slope and bending at the chosen point; those three conditions single out exactly one parabola.

Read alongside