A Guess With Knobs On It

A model is a formula with adjustable numbers in it, and separating those numbers from the data is the distinction the whole subject rests on.

Two kinds of number

Here is a claim about houses: the price is about three thousand pounds per

square metre, plus forty thousand.

That sentence is a model. Give it a house of eighty square metres and it

predicts two hundred and eighty thousand. Give it a different house and it

predicts something else. The size changes from house to house; the three

thousand and the forty thousand do not.

That is the distinction this whole course turns on. There are numbers that

come from the example in front of you, and numbers that belong to the model

and are the same for every example. The first kind are the inputs. The second

kind are the knobs, and fitting a model means choosing them.

FIG 1The model, with the two kinds of number marked

: the input, which comes from the example and changes with every one
: the prediction, which the model produces
: a knob: how much the prediction changes per unit of input
: a knob: the prediction when the input is zero

Two symbols on the right and only one of them is data. The hat on the prediction is there to keep it apart from the true value, which the next lesson needs and which is easy to lose track of once there are several things called y.

Nothing about this is specific to houses or to straight lines. A model with

seven hundred thousand million knobs has exactly the same division: some

numbers arrive with the example and the rest were chosen during fitting and do

not change afterwards. Keeping that clear from the first lesson is worth more

than it sounds, because almost every confusion later on turns out to be a case

of one of them being mistaken for the other.

Reading the two knobs

The two numbers in a straight line are unusually readable, and it is worth

spending a moment on them because larger models forfeit this and it is the

main thing they forfeit.

The offset is what the model predicts when the input is zero. Sometimes that

is meaningful, such as a standing charge on a bill. Sometimes it is nonsense,

such as the price of a house of zero square metres, and in that case it is

still doing a job: it positions the line vertically so the slope can be right

over the range where the data actually lives.

The slope is how much the prediction changes for each additional unit of

input. That is a statement about the world and it carries units: pounds per

square metre, minutes per kilometre, degrees per hour. If the slope of a

fitted model does not have sensible units, something has gone wrong before any

arithmetic.

FIG 2What the slope does, with the offset held still

ww = 3000

prediction

Turning the slope pivots the line about its crossing point on the left and leaves the offset where it is. The two knobs do genuinely separate things, which is why a fitted line can be wrong in two distinguishable ways: wrong height and wrong tilt.

What the shape forbids

Choosing a straight line is a decision, and it is a larger decision than

choosing the two numbers, because the numbers will be fitted automatically and

the shape will not.

A straight line says that each extra unit of input changes the prediction by

the same amount, wherever you are. That is often roughly true and it is

sometimes badly false. Doubling the size of a house does not double its price

in most markets. The effect of an extra hour of study is not the same for the

first hour as for the twentieth. A dose response is frequently flat, then

steep, then flat again, and no straight line of any slope or offset resembles

that at all.

FIG 3Three relationships, one of which a line can represent

a straight relationshipflat, then steep, then flataccelerating

The best straight line through each of the lower two curves is a real line with real numbers, and it is wrong in a patterned way: too high in the middle and too low at both ends, or the reverse. That pattern in the errors is the symptom, and recognising it is the main skill this lesson is preparing.

This matters more than it appears, because fitting always succeeds. Hand a

straight line to data of any shape and it will produce two numbers. Nothing

complains. The model does not know it is the wrong shape, and the score from

the next lesson will be a single number that gives no hint of which kind of

wrongness it represents.

The defence is to look at where the model is wrong rather than only at how

wrong it is. Errors scattered randomly above and below mean the shape is

defensible. Errors that are all positive in the middle and all negative at the

ends mean the shape is wrong, and no amount of fitting will fix it.

FIG 4Four data sets with the same fitted line

	fitted slope	fitted offset	is a line the right shap
roughly straight	500	3000	1
curved	500	3000	0
straight with one outlie	500	3000	0
one point doing all the	500	3000	0

These four sets of points produce the same two numbers to several decimal places and look nothing like each other when drawn. The columns are in hundredths, so the fitted slope is five and the offset thirty. Only the first is a case where a line is a reasonable description, and no summary number distinguishes them.

What fitting will mean

So far there is a formula with two free numbers and no way to choose them. Any

pair is a model; most of them are bad.

Fitting means choosing the knobs, and that requires something this lesson does

not yet have: a rule saying which setting is better than which. Without it the

word better has no content, and with it everything else in the course follows

mechanically.

FIG 5The model, and the fact that nothing yet chooses the knobs

python

def predict(x, w, b):
    return w * x + b  # two knobs, one input
sizes = [52, 61, 75, 80, 94, 110]
prices = [196, 230, 265, 290, 330, 385]  # thousands
for w, b in [(3.0, 40), (3.5, 10), (2.0, 120)]:
    guesses = [round(predict(x, w, b)) for x in sizes]
    print("w", w, "b", b, "gives", guesses)
print("three settings, three sets of predictions, and so far no reason to prefer any of them")

All three settings are models and all three produce predictions. Reading the numbers, the first looks closest, but looks is not a criterion and will not scale to a model with a million knobs. The next lesson replaces looking with a single number.

Six data points and two knobs is a deliberately small situation, and every

lesson up to the fifth will stay at about that size, because the point is to

do the arithmetic rather than to read about it. Everything here is the same

machinery that runs on a million examples; only the patience required is

different.

Recap

A model is a formula containing numbers that are free to be chosen, and fitting a model means choosing them.
The data goes into the formula and the knobs stay with the model, which is the difference between what varies per example and what is shared across all of them.
The shape of the formula is a decision made in advance, and no amount of turning the knobs can produce a shape the formula cannot express.

This is the reading half

Starting the course gives you your own copy of it. Every idea on every page has problems standing under it, marked with a reason rather than a tick, and any sentence you do not believe can be opened and argued with. None of that can happen on a page nobody owns.

The contents

NextMeasuring How Wrong It Is →