One Tree Is Unstable
Last timeKnowing When to Stop Growing
Change a handful of training rows and a tree can come out looking entirely different, which makes a single tree untrustworthy and is also the opening the rest of the course walks through.
Remove two rows and look again
Take the twelve applications, grow a tree, write it down. Now remove two rows
at random, grow again, and compare.
Frequently the two trees share nothing but their first question, and sometimes
not even that. If two candidate questions scored 0.210 and 0.203, removing the
right two rows swaps them, and once a different question sits at the top every
group below it is different, so every question below it is chosen on different
data.
| step | run | question at the top | second question on the yes side | accuracy on held back rows |
|---|---|---|---|---|
| 1 | 1 | income above 30 | has other debts | 0.81 |
| 2 | 2 | income above 30 | age above 44 | 0.79 |
| 3 | 3 | has other debts | income above 28 | 0.80 |
| 4 | 4 | income above 31 | lives in a city | 0.78 |
| 5 | 5 | has other debts | income above 33 | 0.82 |
Why the flipping happens
The choice at each node is winner takes all. Two questions scoring 0.229 and
0.221 are, for practical purposes, equally good, and the procedure does not
record that. It takes the first and discards the second entirely.
So the tree magnifies a difference that was never meaningful. A gap of seven
thousandths in a quantity estimated from twelve rows is noise, and the
procedure converts it into a completely different model. Worse, the effect
compounds: a different question at the top means different groups below,
which means different comparisons, which means different winners, all the way
down.
Models in the previous course do not behave this way. Remove two rows from a
straight line fit and the knobs move slightly. There is no winner takes all
comparison anywhere in that method, so there is nothing to flip.
The lesson stops here
5 more paragraphs to go
You have read the opening. The rest of the argument, the problems that check whether it landed, and the lines worth keeping at the end all come with a plan.
The first lesson of every course in the library reads the whole way through, free, so you can see exactly what the rest of them are.
See the planThe contentsThis is the reading half
Starting the course gives you your own copy of it. Every idea on every page has problems standing under it, marked with a reason rather than a tick, and any sentence you do not believe can be opened and argued with. None of that can happen on a page nobody owns.
The contents