Knowing When to Stop Growing
Last timeGrowing a Whole Tree
A tree grown until every leaf is pure memorises its training data exactly, so growth has to be limited, and cutting back after growing works better than stopping early.
The tree that remembers everything
Remove the stopping rules from the growing procedure and it keeps splitting
until every leaf is pure. On data with no exactly duplicated rows, that means
most leaves hold a single training row.
Such a tree predicts every training row perfectly. Its training score is one
hundred percent, every time, on any data set. The score is therefore not a
measurement of anything: it is a property of the procedure.
| leaves | correct on the training | correct on held back row | |
|---|---|---|---|
| depth 1 | 2 | 74 | 72 |
| depth 3 | 8 | 86 | 84 |
| depth 6 | 41 | 97 | 79 |
| grown without limit | 312 | 100 | 71 |
What has happened is worth saying precisely. The tree has not learned a bad
relationship. It has learned no relationship: each leaf is a record of one
observation, and a new case is answered by whichever training row happens to
be nearest in the particular sequence of questions the tree asked. That is a
lookup table with extra steps.
Three limits
Growth is limited in three ways, and implementations offer all three.
A maximum depth caps how many questions may be asked in a row. It is blunt,
because it limits every branch equally whether or not each branch needs it,
and it is the easiest to reason about.
A minimum number of rows in a leaf refuses to create a leaf resting on fewer
than that many observations. This is the most directly meaningful of the
three, because it is a statement about evidence: a prediction must be
supported by at least this many cases before the model will make it.
A minimum improvement refuses a split whose value, in the sense of the first
lesson, is below a threshold. It stops the tree spending depth on questions
that barely clean anything up.
The lesson stops here
3 more paragraphs to go
You have read the opening. The rest of the argument, the problems that check whether it landed, and the lines worth keeping at the end all come with a plan.
The first lesson of every course in the library reads the whole way through, free, so you can see exactly what the rest of them are.
See the planThe contentsThis is the reading half
Starting the course gives you your own copy of it. Every idea on every page has problems standing under it, marked with a reason rather than a tick, and any sentence you do not believe can be opened and argued with. None of that can happen on a page nobody owns.
The contents