Answers · How models work
Why does changing a few rows of the training data give a completely different tree?
In shortA tree picks the single best split at each step, so a tiny data change can swap a near-tied split and reshape the whole tree; that is normal, not a bug.
A tree is built by choosing, at each node, the single best yes-or-no question among many candidates that are often nearly as good. Removing a couple of rows can swap the winner at the top, and because every later question is chosen inside the groups that top question creates, the whole tree can change.
Here is the arithmetic behind a typical flip. Two candidate questions score and on twelve rows. The winner is the first, but the margin is seven thousandths, and removing two rows removes one sixth of the sample:
Why the natural guess fails
The natural guess is that a model should move continuously when the data move a little. That is what happens in a straight-line fit: the coefficients are chosen by minimising a smooth sum of squared errors, so taking out two rows nudges the line. A tree has no such continuous knob. It compares candidate questions and keeps only the maximum; the second-best score is discarded, not recorded as a close second. A gap of estimated from twelve rows is noise, but the procedure treats it as a decision and converts it into a different model. The same happens with scores of and : eight thousandths is not a real difference at that sample size, yet it is enough to choose one question and throw the other away.
The change compounds
Once the top question changes, the data are partitioned differently. If the first tree asks whether income is above 30 and the second asks whether the applicant has other debts, the yes group in one tree is not the yes group in the other. Every later split is then chosen inside a different group of rows. A different top question therefore changes the comparisons available below it, which changes the next winner, and that repeats down the tree. The instability is structural: it is not that the new tree is less accurate, it is that a different adequate description has been selected.
| quantity | changes when rows are removed | safe to quote |
|---|---|---|
| the question at the top | yes | no |
| the exact thresholds | yes | no |
| which inputs appear somewhere | no | yes |
| the accuracy | no | yes |
| the prediction for a given case | no | yes |
A straight-line fit does not behave this way. Its coefficients are chosen by minimising a smooth sum of squared errors, so removing two rows nudges them a little; no comparison is winner takes all. A tree's split choice is a maximum over candidates. The maximum is a discontinuous function of the scores: one score can cross another by a tiny amount and the selected question jumps. The tree amplifies a difference that the sample cannot support.
What can be trusted
The stability is in the accuracy, not in the structure. Two trees with completely different questions can score about the same on held-back rows. That means the data support the relationship more firmly than they support any one description of it. The honest reading of a single tree is that it is one of many adequate descriptions, not the explanation. If you want a stable statement, report the accuracy and the inputs that keep appearing somewhere in the tree; treat the exact top question and the exact thresholds as sample artefacts.
The flip is not guaranteed. If the best question is far ahead of the second, removing a few rows will not change it. With many columns, however, several questions usually cluster near the best, so the unstable case is common rather than unlucky. Pruning reduces the amount of sensitivity but does not change its kind: a heavily pruned tree can be stable because it is too rigid, which is a different error.
Where this leads is not a better single tree but an average of many. If each tree is wrong in its own direction because of the rows it saw, averaging partly cancels those directions:
The first term is what averaging cannot remove; the second shrinks as grows. Averaging a hundred straight lines fitted to curved data still gives a straight line, just as wrong as any one of them. Averaging unstable trees is different because their structures disagree, and that disagreement is what cancels.
Worth keeping
- Two trees with very different structures can have almost the same accuracy.
- A single tree's exact splits are less trustworthy than its accuracy or the inputs it uses.
- Averaging many unstable trees cancels part of their sample-specific error; averaging stable wrong-shaped models does not.
Where this goes next
This question is one step of Trees, and Committees of Them, a course that works through its subject from the beginning. The page that teaches this part of it is One Tree Is Unstable, and it reads the whole way through whether or not you have an account.
What an account changes is the practice: problems standing under every idea, marked with a reason rather than a tick, in a copy of the course that is yours.
Read that pageThe whole courseOther questions people ask about this
- Why do tree ensembles still beat neural networks on ordinary spreadsheet data?Because each spreadsheet column means something on its own, which is exactly what a one-column-at-a-time split needs; where inputs only mean something in combination, networks win by tens of points.
- How does a decision tree pick the exact number to split a column on?A decision tree sorts the column, tries a cutoff between every adjacent pair of distinct values, scores each one by how mixed the two resulting groups are, and keeps the best.
- What is Gini impurity actually measuring?Gini impurity is the chance that two rows picked independently from a group have different outcomes. It is zero for a pure group and largest for an even split.
- Why does a tree that gets every training example right do worse on new data?A tree that fits every training row has memorised individual cases rather than learned a pattern, so its accuracy on new rows falls.
- Why does the model get stuck repeating the same sentence over and over?The model is not stuck in a memory loop: once it has written a phrase, that phrase is in its context and makes writing it again more likely, so a decoder that always picks the top word cannot leave.
- Why does diffusion add Gaussian noise instead of any random corruption?Diffusion adds Gaussian noise because Gaussian errors combine exactly: each step keeps the spread fixed and gives a closed-form noisy picture at any later step, which makes the reverse step learnable.