Answers · How models work

How many principal components should I keep?

In shortThere is no single correct number: keep the smallest number whose combined variation you can afford to lose, and report the share you discarded.

There is no correct number. You choose how much variation you can afford to lose, then keep the smallest number of directions whose combined share leaves you with the rest. The method reports that share, so the choice is a stated cost rather than a guess about the data's true dimension.

Put the columns on a common footing first, usually by centring each column and dividing by its standard deviation. Without that step a column measured in pounds, or a count with a large range, can dominate the first direction simply because its numbers are larger. Once the columns are on the same scale, the first direction is the unit length mixture $w_1$ that maximises the average squared score $\frac{1}{n}\sum_i (w_1^{\top} x_i)^2$ . The second maximises the same quantity subject to being at right angles to the first, and later directions repeat that constraint against all earlier ones. The share carried by a direction is its maximised quantity divided by the total, and the shares add to one. That sum is what makes the reduction measurable rather than hopeful.

41+23=64,\quad 64+12=76,\quad 76+6=82,\quad 82+4=86,\quad 86+2=88,\quad 88+2=90,\quad 90+1=91.

The arithmetic of a cutoff

On a sixty column customer table whose first eight directions carry 41, 23, 12, 6, 4, 2, 2 and 1 per cent, the running totals are as above. An eighty per cent target is first crossed by the fourth direction: 82 per cent kept, so 100-82=18 per cent discarded. A ninety per cent target needs seven directions: 90 kept, so 10 discarded. Three directions keep 76 and discard 24. Five keep 86 and discard 14. Eight keep 91 and discard 9. Those are different answers to the same question, and each is exact about its cost. The fourth direction is not a natural break; it is the first one that happens to pass 80.

FIG 1

directions kept	share kept (%)	share discarded (%)
1	41	59
2	64	36
3	76	24
4	82	18
5	86	14
6	88	12
7	90	10
8	91	9

Three rules, none of them correct

Three rules are defensible, and none is the correct one. Keep enough to reach a stated share, commonly eighty or ninety per cent. Keep the directions before the curve of individual shares flattens. Or keep two or three because the purpose is a picture, which is honest when a picture is genuinely the purpose. The first rule is the easiest to audit because the threshold is stated in advance. The second is weaker: a bend is a visual judgement, and a smooth fall has no bend. The third is not a statistical rule at all, but it is not a statistical problem either if two numbers are all you can plot. In every case, report the number kept and the share discarded together.

Why the natural guess fails

The natural guess is that the data contains a right number of components and the job is to find it. The shares do not contain a cutoff. They measure what each direction carries, not which direction is the last true one. A sharp bend can suggest a stopping point, but if the shares fall smoothly from 41 to 23 to 12 to 6 to 4, there is no sharp bend at all, and choosing where to stop is still choosing a cost. Keeping enough for ninety per cent is a convention, not a discovery. It is a good convention because it is explicit, but it is not more correct than keeping enough for eighty, or for a picture, provided you say which you did.

\text{independent columns: each direction carries } \frac{1}{60}\approx 1.67\%,\quad \text{keeping 20 keeps } \frac{20}{60}=\frac13\approx 33\%.

Two assumptions change the answer. If the columns are genuinely independent, every direction carries about one sixtieth and a flat climb is the honest result: keeping twenty of sixty keeps a third of the variation, so the reduction has gained nothing. If the small directions contain a rare fault, discarding them can discard the fault, and the reported share will look excellent while the thing you needed was removed. Reduce before flagging rare faults and you can hide them. Reduce before drawing a wide table and you can plot it. Reduce before feeding a model that must be explained and you inherit mixtures that nobody can justify. The purpose decides whether the discarded variation is noise or evidence.

A direction is a mixture of columns, not a column. It was chosen to spread the data out, so calling the first customer direction engagement is a hypothesis about the mixture, not a measured quantity. Name it, list the columns with large weights, and test the name against something outside the reduction. That does not change how many to keep, but it changes what you are allowed to say about them once kept.

Worth keeping

Keeping four directions from a table whose first four carry 82 per cent of the variation discards 18 per cent, and that discarded amount is the price, not a rounding error.
If sixty columns are independent, each direction carries 1/60 of the variation and no small set of directions can replace them.
A direction chosen to maximise spread has no meaning built into it, so any name attached to it is a hypothesis to be tested.

Where this goes next

This question is one step of Finding Structure Without Labels, a course that works through its subject from the beginning. The page that teaches this part of it is Describing It With Fewer Numbers, and it reads the whole way through whether or not you have an account.

What an account changes is the practice: problems standing under every idea, marked with a reason rather than a tick, in a copy of the course that is yours.

Read that page The whole course