Working Without an Answer Key

Everything in the previous two courses rested on one arrangement. Each row

came with an answer, some rows were hidden, the model was asked about them,

and the comparison was the score.

Now the answers are gone. A table of forty thousand customers, each with

sixty columns and no column saying what they are. A pile of documents. A month

of sensor readings. Nobody has labelled any of it, and labelling it would cost

more than the project is worth.

This is the ordinary situation. Most data that exists has never been

labelled, and the methods for it are the easiest in the field to fool yourself

with, for a reason that this lesson is entirely about.

Naming which of the three a task is settles a great deal before any method is

chosen. A request to understand the customers is not a task; a request to

divide them into a handful of groups that marketing will treat differently is

one, and it carries its own standard of success, which is whether marketing

can act on the groups.

In the previous courses a model was judged by hiding some answers and checking

against them. That procedure needs answers. Here there are none, so there is

nothing to hide and nothing to check.

This is not a gap that cleverness closes. It is the definition of the

situation. Several numbers get reported as though they were scores, and every

one of them is computed from the same data the method was fitted on, using the

same notion of distance the method was trying to optimise. A method that

groups points tightly around centres will score well on a measure of how

tightly points sit around centres, whatever the data looks like.

Here is the fact that makes this subject dangerous.

Ask a grouping method for four groups and it returns four groups. It returns

them on data with four real groups in it, and it returns them on data drawn

uniformly from a square, with no structure of any kind. The output looks the

same: four centres, four coloured regions, a chart anybody would accept.

The procedure has no way to report that the question was wrong, because it was

never given a way to represent that answer. It was asked for four groups and

four groups is what it knows how to produce.

Two things cannot be manufactured by the method itself, and between them they

carry most of the honest evidence available in this course.

The first is stability. Refit the method on a different sample of the same

data, or on nine tenths of it, and see whether roughly the same structure

comes back. A real division of the data survives this. A division the method

imposed on an empty square does not, because there was nothing determining

where the boundaries fell.

The second is prediction of something held out. Find a column that was not

used in the grouping, something known about each row but kept out of the

method, and ask whether the groups differ on it. Groups built from purchase

history that turn out to differ sharply in cancellation rate have told you

something, and the cancellation rate was never available to the method.

The rest of the course teaches the methods and this scepticism together,

because teaching them apart produces people who can run a grouping and cannot

tell whether it means anything. The next lesson starts underneath all of them,

with the notion of distance that every one of these methods is built on and

that quietly decides most of the answer.