Measuring How Wrong It Is

Last timeA Guess With Knobs On It

Fitting needs a single number saying how bad a setting of the knobs is, and the choice of that number is a decision about which mistakes matter.

Why one number

The previous lesson ended with three settings of the knobs and no way to

choose between them. Looking at the predictions and judging works at six

points and two knobs, and stops working immediately after that.

The problem is that each setting produces a whole set of errors, one per

example, and sets of numbers cannot be ordered. One setting might be closer on

four houses and further on two. Which is better? The question has no answer

until the sets are collapsed into one number each, and then it has an obvious

one.

So the first requirement is a score: a single number computed from all the

predictions and all the truths, which is smaller when the model is doing

better. Everything else in fitting is the search for the knobs that make it

small.

The errors, and what to do with them

For each example there is an error, which is the prediction minus the truth.

It is positive when the model guessed too high and negative when it guessed

too low.

Adding the errors up directly does not work, and the reason is immediate: a

model that is a hundred too high on one house and a hundred too low on the

next would score zero, which is not a description of a good model. The signs

have to go before anything is added.

There are two obvious ways to remove a sign. Take the size of each error and

ignore its direction, or square each error. Both work. They are different

scores, they prefer different models, and the difference is not small.

FIG 1The two scores

: the prediction for example i
: the true value for example i
: the number of examples

Both are averages over the examples, so the score does not grow simply because there is more data. The difference is what happens to one error of ten against ten errors of one: the left score says the first is ten times worse, and the right score says they are the same.

What squaring decides

Squaring is usually presented as a convenience, and it is convenient, but the

substantive thing it does is decide how much large errors matter.

An error of two contributes four. An error of ten contributes a hundred. So

the squared score considers one error of ten to be worse than twenty five

errors of two, and will happily make twenty five predictions slightly worse to

pull one prediction closer. The plain size score does the opposite: it counts

an error of ten as worth exactly ten errors of one, and it will leave a single

far away point alone rather than disturb everything else.

The lesson stops here

4 more paragraphs to go

You have read the opening. The rest of the argument, the problems that check whether it landed, and the lines worth keeping at the end all come with a plan.

The first lesson of every course in the library reads the whole way through, free, so you can see exactly what the rest of them are.

See the plan The contents

This is the reading half

Starting the course gives you your own copy of it. Every idea on every page has problems standing under it, marked with a reason rather than a tick, and any sentence you do not believe can be opened and argued with. None of that can happen on a page nobody owns.

The contents