Measuring How Wrong a Straight Line Is, on Twelve Points

Twelve measurements arrive, one output for each input from one to twelve, and

the question is which straight line fits them. As it stands the question cannot

be answered, because fitting is not a quantity. Two people can draw different

lines through the same twelve points, both be satisfied, and have no way of

arguing with each other.

So the first piece of work is not a line. It is a number, a score that any

candidate line can be fed into, which comes out larger when the line misses by

more. Once that number exists the question is arithmetic, and everything built

later is machinery for making it small.

Write $x_i$ for the $i$th input and $y_i$ for the measurement taken at it, with

$i$ running from 1 to $n$ and $n = 12$. These twelve pairs are the only data in

play, and every number below is computed from them.

Five quantities built out of the table get used again and again, so they are

worth computing once and carefully. The inputs add to

$1 + 2 + \cdots + 12 = 78$. Their squares add to

$1 + 4 + 9 + 16 + 25 + 36 + 49 + 64 + 81 + 100 + 121 + 144 = 650$. The

measurements add to

$6 + 7 + 10 + 15 + 18 + 19 + 22 + 27 + 30 + 31 + 34 + 39 = 258$. The twelve

products $x_i y_i$ are 6, 14, 30, 60, 90, 114, 154, 216, 270, 310, 374 and 468,

and they add to 2,106. Dividing the two plain sums by twelve gives the mean

input $\bar{x} = 78/12 = 6.5$ and the mean measurement

$\bar{y} = 258/12 = 21.5$. Those five sums are the whole of what is ever needed

about the twelve points, since two different sets of measurements sharing them

are scored identically by everything below, and that is the first hint that the

method is cheap: a thirteenth measurement costs five additions and no further

work at all.

A candidate line is two numbers. Write $w$ for its slope and $b$ for its

height where the input is zero, so the line's prediction at input $x$ is

$w x + b$. The prediction at the $i$th input is written $\hat{y}_i = w x_i + b$,

and the amount the line misses that measurement by is the residual

The residual is signed. It is positive where the measurement sits above the

line and negative where it sits below, and that sign is the reason the twelve

residuals cannot simply be added. Take the flat line $w = 0$, $b = 21.5$, which

sits at the height of the mean measurement. Its residuals are

$6 - 21.5 = -15.5$ at the first point and $39 - 21.5 = 17.5$ at the last, and

they add to $258 - 12 \times 21.5 = 258 - 258 = 0$ exactly. A line that is

17.5 away from one of the points has scored zero on the raw sum. The squared

total for that same flat line is 1,299, computed below, which is over a hundred

times the best line's score, so the raw sum of residuals is not a weak measure

of wrongness but a measure of something else entirely.

Destroying the sign before adding is therefore compulsory, and squaring does

it.

The squared total for the flat line can now be done properly. The squares of

the measurements add to

$36 + 49 + 100 + 225 + 324 + 361 + 484 + 729 + 900 + 961 + 1156 + 1521 = 6846$,

and expanding each squared residual gives

$E(0, 21.5) = \sum_i (y_i - 21.5)^2 = 6846 - 2 \times 21.5 \times 258 + 12 \times 21.5^2$,

which is $6846 - 11094 + 5547 = 1299$.

Two more names, both needed later. The mean squared error is $E/n$, the

total divided by twelve, which makes scores comparable when the number of

points changes. Its square root is the typical size of a single miss, in the

same units as the measurements.

The first candidate is $w = 3$, $b = 2$, the line $y = 3x + 2$. The second is

drawn the way a person with a ruler draws one, through the first and last

measurements. Its slope is $(39 - 6)/(12 - 1) = 33/11 = 3$, and matching the

first point gives $6 = 3 \times 1 + b$, so $b = 3$ and the line is

$y = 3x + 3$. It is a defensible candidate, and it turns out to pass exactly

through six of the twelve measurements rather than only two.

The arithmetic is worth seeing written out, because it is the whole of the

score. For $y = 3x + 2$ every squared residual is $(\pm 1)^2 = 1$ and there are

twelve of them, so $E(3, 2) = 12$ and the mean squared error is

$12/12 = 1$. For $y = 3x + 3$ six residuals are zero and six are $-2$, so

$E(3, 3) = 6 \times 0 + 6 \times 4 = 24$ and the mean squared error is

$24/12 = 2$. The typical miss is 1 for the first line and

$\sqrt{2} \approx 1.41$ for the second.

The two candidates share a slope and differ only in their intercept, which

makes the comparison sharper than it looks. Moving the intercept from 2 up to

$2 + c$ subtracts $c$ from every residual at once, so

using the two facts that the first line's squared residuals add to 12 and its

raw residuals add to zero. At $c = 1$ that gives $12 + 12 = 24$, which agrees

with the point by point total. One unit of intercept costs 12, and the cost

grows as the square of the error in the intercept rather than in proportion

to it.

Nothing above forced the square. Taking absolute values destroys the sign just

as well, so it has to be argued rather than assumed, and the argument is not

all one way.

Score both candidates by total absolute error. The first line has twelve

residuals of size one, giving $12 \times 1 = 12$. The second has six of size

zero and six of size two, giving $6 \times 0 + 6 \times 2 = 12$. The two lines

tie. The measure that was supposed to choose between them has no opinion, and

the tie is not a coincidence of these two candidates.

The flat stretch can be read straight off the residuals. With the slope at 3,

six residuals are $+1$ and six are $-1$, so moving the intercept to $2 + c$

makes the absolute total $6|1 - c| + 6|1 + c|$, which for any $c$ between $-1$

and $1$ is $6(1 - c) + 6(1 + c) = 12$, with the $c$ terms cancelling. The

absolute total is constant across that whole interval, and it starts to rise

only once the intercept leaves it.

So squaring buys three things at once. The score is smooth, which absolute

error is not at zero. The score has one lowest point rather than a valley of

equally good answers. And the derivative of a square is linear in the unknowns,

which is what lets the best line be solved for rather than searched for.

What squaring costs is proportion, and the cost is easiest to see in a single

bad reading. Suppose the twelfth measurement were mistyped as 51 instead of 39.

The line $y = 3x + 2$ predicts 38 there, so that residual becomes

$51 - 38 = 13$ and its square is 169, while the other eleven squared residuals

are still 1 each.

The honest summary is that squaring is chosen for its arithmetic and for one

statistical reason, not because it is the truest account of wrongness. The

statistical reason is worth stating precisely: if each measurement is the truth

plus an independent error drawn from the same normal distribution, then the

line minimising the squared total is the line under which the observed

measurements are most likely. Change that assumption about the errors and a

different score becomes the right one, which is why absolute error and its

relatives are reached for when a few readings are expected to be wild.

A score now exists, two candidates have been fed through it, and one of them is

twice as good as the other. That is all. Nothing above shows that $y = 3x + 2$

is the best line, only that it beats a particular rival, and there is a

continuum of candidates that have not been tried.

What makes the next step possible is the shape of $E$ rather than any cleverness

about lines. It is a smooth function of two numbers with one lowest point, and

at that lowest point its two partial derivatives are both zero. Writing those

two conditions out, with the five sums from the table in place of the

measurements, gives two equations in $w$ and $b$ that can be solved exactly.