Answers · The mathematics underneath

Why does adding a penalty on the weights stop overfitting?

In shortAdding a penalty makes large weights costly, so the fit keeps them only when the training data saves enough error to pay for them, which limits the settings the model can reach.

A weight penalty stops overfitting by making large weights expensive, not impossible. The fit minimises training error plus a charge for size, so a large-weight answer survives only when its extra fit is worth more than its price; that shrinks the set of settings the run actually reaches and makes the fitted function less sensitive to the particular training sample.

\mathcal{L}(\theta) = \frac{1}{n}\sum_{i=1}^{n} \ell\!\left(y_i, f_\theta(x_i)\right) + \lambda\,\Omega(\theta)

Read the plus sign as a price list. Nothing has been forbidden: a large weight is still available and will still be chosen if it lowers the training error by more than the charge it adds. The charge $\Omega(\theta)$ grows with the size of the settings, and $\lambda$ sets how strong the charge is. Raising $\lambda$ does not make the model smaller in the sense of removing functions from the family. It makes the procedure prefer the smallest answer consistent with the data, which is a smaller reachable set.

The arithmetic of insistence

Take two candidate settings on the same data. The small setting has training error 0.30 and size measure 1. The large setting has training error 0.20 and size measure 9. With no charge, the large setting wins because 0.20 is less than 0.30. Add a charge of strength $\lambda$ and the totals are $0.30+\lambda$ and $0.20+9\lambda$ . The large setting wins only while $0.20+9\lambda < 0.30+\lambda$ , which rearranges to $8\lambda < 0.10$ , so $\lambda < 0.0125$ . At $\lambda=0.0125$ the two tie at 0.3125. Above that the small setting wins. The data has to insist on the large setting by saving more than the charge costs.

FIG 1

charge strength λ	small setting: 0.30 + λ × 1	large setting: 0.20 + λ × 9	lower total
0	0.30	0.20	large
0.005	0.305	0.245	large
0.0125	0.3125	0.3125	tie
0.02	0.32	0.38	small
0.1	0.40	1.10	small

The gradient picture matches the arithmetic. For $\Omega(\theta)=\sum_j \theta_j^2$ the added gradient is $2\lambda\theta_j$ , so at $\lambda=0.05$ a weight of 4 feels a pull of $2\times0.05\times4=0.4$ while a weight of 0.5 feels $2\times0.05\times0.5=0.05$ . Large weights shrink most and small ones barely move. For $\Omega(\theta)=\sum_j |\theta_j|$ the pull has the same size at every scale, so small weights are pushed all the way to exactly zero and stay there. In both cases the charge removes high-weight answers from the set the fit will actually reach unless the training error pays for them.

What the natural guess gets wrong

The natural guess is that a penalty forces weights to be small, and small weights are simple, so a simple model cannot overfit. That gets the direction right and the mechanism wrong. A penalty does not know which weights are useful. It charges every large weight, including a large weight that is the only way to capture a real effect. If the data saves more error than the charge costs, the large weight stays. If the charge is too weak, nothing is removed and overfitting continues. If the charge is too strong, useful large weights are removed as well and the model underfits. The penalty is not a simplicity knob that can be turned up for safety. It is a price, and the right price depends on how much the data can afford.

The two classical charges differ in character. Sum of squares pulls each weight towards zero in proportion to its current size, so large weights shrink a lot and small ones a little, and no weight ever arrives exactly at zero. Correlated inputs end up sharing the available weight. Sum of absolute values pulls with the same force regardless of how small a weight already is, so small weights reach exactly zero. That gives selection as well as shrinkage. Neither charge is more correct in general: one suits a problem where many small contributions are expected, the other where only a few inputs matter.

Where the charge stops helping

The strength $\lambda$ cannot be chosen by training error, because adding a charge can only make training error worse, so the training objective always prefers the weakest charge available. It must be chosen against held-out examples: try several strengths, watch held-out error, and keep the one at the minimum. Turn the dial too far down and sensitivity still dominates. Turn it too far up and rigidity dominates, with training error and held-out error both rising. Both ends are failures and they look different. The strength is itself a quantity fitted to held-out data, so if the result is to be reported honestly a third sealed set is needed. More data is the other lever, and it reduces sensitivity without adding rigidity.

Worth keeping

A penalty does not forbid a large weight; it puts a price on it, and the training data must reduce the error by more than that price for the large weight to survive.
Choosing the penalty strength against training error always picks the weakest penalty, so the strength must be chosen against held-out examples.
Push the penalty too far and the model stops being flexible enough, so held-out error rises again.

Where this goes next

This question is one step of Why Learning Works At All, a course that works through its subject from the beginning. The page that teaches this part of it is Making Some Answers Harder To Reach, and it reads the whole way through whether or not you have an account.

What an account changes is the practice: problems standing under every idea, marked with a reason rather than a tick, in a copy of the course that is yours.

Read that page The whole course