Basis Lab

Answers · How models work

Why does linear regression square the errors instead of just adding their absolute values?

In shortSquaring makes large errors count much more than small ones, so it picks a different line from absolute value; absolute value is a different choice about costs, not a mistake.

Squaring the errors is not what makes them positive; absolute value does that too. Squaring is what makes one large error count for more than several small ones, and that changes which straight line wins.

Lsq=1n∑i=1n(y^i−yi)2Labs=1n∑i=1n∣y^i−yi∣L_{\text{sq}} = \frac{1}{n}\sum_{i=1}^{n}\big(\hat{y}_i - y_i\big)^2 \qquad\qquad L_{\text{abs}} = \frac{1}{n}\sum_{i=1}^{n}\big|\hat{y}_i - y_i\big|

Take five errors. If they are 2,2,2,2,22,2,2,2,2, the plain size score is (2+2+2+2+2)/5=10/5=2(2+2+2+2+2)/5 = 10/5 = 2, and the squared score is (4+4+4+4+4)/5=20/5=4(4+4+4+4+4)/5 = 20/5 = 4. If they are 0,0,0,0,100,0,0,0,10, the plain size score is (0+0+0+0+10)/5=10/5=2(0+0+0+0+10)/5 = 10/5 = 2, and the squared score is (0+0+0+0+100)/5=100/5=20(0+0+0+0+100)/5 = 100/5 = 20. The plain size score cannot tell those two sets apart. The squared score prefers the first by a factor of five, because 20/4=520/4 = 5. A third set, 1,1,1,1,61,1,1,1,6, has plain size score (1+1+1+1+6)/5=10/5=2(1+1+1+1+6)/5 = 10/5 = 2 and squared score (1+1+1+1+36)/5=40/5=8(1+1+1+1+36)/5 = 40/5 = 8.

What the square is buying

An error of 22 contributes 22=42^2=4 under the squared score but ∣2∣=2|2|=2 under the plain size score. An error of 1010 contributes 102=10010^2=100 under the squared score but ∣10∣=10|10|=10 under the plain size score. The squared score charges 100/4=25100/4=25 times as much for the ten as for the two; the plain size score charges 10/2=510/2=5 times as much. So a squared-score fit will accept several small errors to pull one big error down, while a plain-size fit treats ten errors of one as exactly as bad as one error of ten.

FIG 1
errorssquared scoreplain size score
2, 2, 2, 2, 242
0, 0, 0, 0, 10202
1, 1, 1, 1, 682

The natural guess is that absolute value is the obvious way to make errors positive, so squaring must be a convenience or a trick. Sign removal is the easy part. The hard part is the trade-off after the signs are gone: the score has to say whether one miss of ten is worse than five misses of two, and by how much. Absolute value says error cost grows in proportion to size. Squaring says it grows with the square of size. Neither is the unique correct way to score a fit; they are different claims about what a bad miss costs.

The disagreement is not small. With the same plain size score of 22, the squared score can be 44, 88, or 2020 depending on how the error is distributed. The fit that wins under one score need not win under the other. A line that leaves one point far away and fits the rest almost exactly can look good to the plain size score and bad to the squared score, because the squared score multiplies that one far error by itself. A line that spreads small errors over every point can look good to the squared score and only average to the plain size score. That is the whole content of the choice.

Smoothness is a separate, practical reason for the default. At an error of 00, ∣e∣|e| has no derivative: from the left the slope is −1-1, from the right it is +1+1. The function e2e^2 has derivative 2e2e, which is 00 at e=0e=0 and exists everywhere. This matters because the usual fitting method asks which way is downhill. A kink at zero is a point where that question has no single answer. You can still fit with absolute value, by other methods or by smoothing the kink, but the smooth squared score is the easier first case.

When the square is the wrong story

Squaring is the default for three concrete reasons. It gives a straight-line fit an exact solution; it is smooth everywhere, so its derivative exists at every error including zero; and it matches the assumption that errors are roughly normal. But if a large error is a measurement mistake or a genuine oddity that should not drag the line, squaring will overreact to it. If a large error is disproportionately costly, such as a delivery missing by an hour rather than four times by fifteen minutes, squaring describes the cost better. The choice of score is therefore a statement about which mistakes matter, not a theorem that one score is right.

The score is not a measure of how good the model is in any absolute sense. It is a way to rank settings of the knobs, and changing from plain size to square can change the ranking even when both scores are finite and both remove the sign. Differentiability is a real reason for the default, but it is not the reason the two scores disagree; they disagree because they charge different prices for the same error.

Worth keeping

  • Squaring and absolute value both remove the sign, but they disagree about how much a large error should count.
  • A score only ranks settings of the knobs; it is not an absolute measure of model quality.
  • Squaring is convenient because it is smooth and has an exact solution, not because absolute value is invalid.

Where this goes next

This question is one step of The Simplest Model That Learns, a course that works through its subject from the beginning. The page that teaches this part of it is Measuring How Wrong It Is, and it reads the whole way through whether or not you have an account.

What an account changes is the practice: problems standing under every idea, marked with a reason rather than a tick, in a copy of the course that is yours.

Read that pageThe whole course

Other questions people ask about this

  1. What is actually new in a neural network compared with linear regression?Only three things are new: the slopes are computed by working backwards, the score is no longer a bowl, and the model builds its own inputs. The prediction, the loss, the step and the held out check are unchanged.
  2. Why does logistic regression use a sigmoid instead of a straight line?The sigmoid turns an unbounded straight-line score into a probability between zero and one, and it pairs with the log score so fitting keeps the same gradient shape.
  3. What is the difference between the inputs and the weights in a linear model?The input is supplied by each example and changes from row to row; the weights are the model's adjustable numbers, chosen once by fitting and shared by every example.
  4. How do I know if I should increase or decrease the slope to reduce the error?The sign of how fast the loss changes with the slope tells you: positive means lower the slope, negative means raise it.
  5. Why does the model get stuck repeating the same sentence over and over?The model is not stuck in a memory loop: once it has written a phrase, that phrase is in its context and makes writing it again more likely, so a decoder that always picks the top word cannot leave.
  6. Why does diffusion add Gaussian noise instead of any random corruption?Diffusion adds Gaussian noise because Gaussian errors combine exactly: each step keeps the spread fixed and gives a closed-form noisy picture at any later step, which makes the reverse step learnable.

Read alongside