Describing It With Fewer Numbers

Last timeGroups That Are Not Blobs

Most wide tables really vary along a handful of directions, and finding those directions replaces sixty columns with three while recording exactly how much was thrown away.

Sixty columns, three real ones

A table of customers has sixty columns. Total spend, spend last month, spend

this month, number of orders, average order value, number of visits, visits

last week, and so on.

Those columns are not sixty separate things. Somebody who spends a lot has

many orders, visits often and has a high total, and knowing any one of them

tells you most of the others. The data has sixty columns and perhaps three or

four actual dimensions of variation, and the remaining fifty-six are repeats

of those three with noise on top.

Finding the real directions is worth doing for three reasons. The distance

problem of the second lesson goes away, because distances in four dimensions

are meaningful where distances in sixty are not. The data can be drawn, since

two or three numbers per row plot and sixty do not. And every other method in

this course runs better afterwards.

A direction is a combination, not a column

The important idea is that the directions are not a selection from the

columns. They are mixtures of them.

Take just height and weight. Both vary a lot, and they vary together: tall

people are heavier. So the data does not fill the two dimensional space at

all; it lies along a diagonal band. One number, the position along that

diagonal, is roughly overall size, and it captures nearly all of the

variation. The second number, the position across the diagonal, is roughly

build, and it captures a little. Neither number is height and neither is

weight.

FIG 1Two columns that really vary along one direction

the direction the data lies alongthe same direction, eight kilograms above itthe same direction, eight kilograms below it

Real people fill the band between the outer two lines rather than scattering over the whole chart. The long direction of the band carries most of the variation and is the first direction; the short direction across it carries the rest and is the second. The method finds the long direction by looking for the line the points sit closest to, which is the same line.

The method, and what it is choosing

Put the columns on a common footing first, since the second lesson applies

here with full force: a column measured in pounds would otherwise define the

first direction by itself.

Then find the direction along which the data spreads out most. Record it.

Then find the direction along which the data spreads out most among the

directions unrelated to the first. Record that. Continue. With sixty columns

there are sixty such directions, and they come out in order of how much

variation each one carries.

The lesson stops here

5 more paragraphs to go

You have read the opening. The rest of the argument, the problems that check whether it landed, and the lines worth keeping at the end all come with a plan.

The first lesson of every course in the library reads the whole way through, free, so you can see exactly what the rest of them are.

See the plan The contents

This is the reading half

Starting the course gives you your own copy of it. Every idea on every page has problems standing under it, marked with a reason rather than a tick, and any sentence you do not believe can be opened and argued with. None of that can happen on a page nobody owns.

The contents