Answers · How models work
How does a decision tree pick the exact number to split a column on?
In shortA decision tree sorts the column, tries a cutoff between every adjacent pair of distinct values, scores each one by how mixed the two resulting groups are, and keeps the best.
Sort the values in the column and put a candidate cutoff halfway between each adjacent pair of distinct values. Score every candidate by the mixedness left in the two groups it creates, and keep the candidate with the smallest weighted mixedness. The exact number is the midpoint of the winning gap, not a value chosen by eye.
The scan
Here is the Gini mixedness of a group with positives and negatives. and are the numbers of rows sent left and right by cutoff , and is the total. A group that is all one label has . A fifty-fifty group has . is the quantity minimised. It has the same shape as the squared spread used for a numeric outcome, with in place of .
| candidate cut | left labels | right labels | weighted Gini W(c) |
|---|---|---|---|
| 1.5 | 0 | 0,1,0,1,0 | 0.400 |
| 2.5 | 0,0 | 1,0,1,0 | 0.333 |
| 3.5 | 0,0,1 | 0,1,0 | 0.444 |
| 4.5 | 0,0,1,0 | 1,0 | 0.417 |
| 5.5 | 0,0,1,0,1 | 0 | 0.400 |
Take the six rows with labels . The parent group has two positives and four negatives, so its Gini is . At cut 2.5 the left group is , with , so . The right group is , with , so . Weighted: . At cut 1.5 the weighted score is , and at cut 3.5 it is . The smallest is at 2.5, so the split sends and left and right. The reduction is .
The table shows why the best cut can sit in an unremarkable place. Cut 4.5 also looks plausible because it separates the last two rows, but its weighted score is , worse than . Cut 5.5 isolates one zero on the right and scores . The winner is not the cut that makes one side pure. Cuts 1.5 and 5.5 each make one side pure and still lose, because the other side is large and mixed. The criterion is weighted, so a pure small group does not compensate for a mixed large one.
The natural guess is to take the mean, the median, or a gap that looks wide. That fails because the score is not monotone in the cutoff. Moving the cut from 2.5 to 3.5 puts on the left, and is a positive, making the left group and the right group . Both groups then have Gini and the weighted score rises from to . A cutoff placed exactly on an observed value is not wrong, but it needs an explicit rule for equality. The midpoint convention avoids relying on that rule without changing any partition.
There are only candidates when the column has distinct values. The score can change only when a row crosses from one side to the other, and that happens at each observed value. Between two adjacent observed values every cutoff gives the same two groups. For values 2 and 3, the cuts 2.1, 2.5 and 2.9 all send every row with value 2 left and every row with value 3 right. The implementation reports 2.5 because it is the midpoint, not because 2.5 has any special meaning in the data. If the column has duplicates, use adjacent distinct values, not adjacent rows.
If two cuts tie, any deterministic rule works. Implementations usually keep the first in column order or the lowest cut. The tie means the data does not distinguish them under the chosen mixedness measure, not that the cutoff is arbitrary. At prediction time the case's value is compared with the stored cut using the same inequality. Because the cut sits in a gap between training values, a new value in that gap can land on either side depending on which midpoint was stored. Two trees grown on the same data with different midpoint conventions can therefore differ on a new case that falls between the same two training values. Using one convention for growing and prediction removes most of that sensitivity.
Where it stops being exact
The scan stops being exact when the candidate list is pruned to quantiles or histogram edges. That is sometimes done for speed on very wide columns, but the best cut in the pruned list need not be the best cut available. A minimum leaf size can also forbid the lowest-scoring cut, so the chosen cut is the best among those that leave enough rows on both sides. For a categorical column a threshold is meaningless; the split is a subset of categories. For a numeric outcome nothing in the scan changes. Replace by , score each cut by the weighted , and let the leaf report its average. The same finite candidate list is used, and the same midpoint convention applies.
Worth keeping
- A decision tree does not guess a cutoff; it scores every midpoint between adjacent distinct values and keeps the one with the lowest weighted mixedness.
- Any number between the same two adjacent observed values produces exactly the same split, so the midpoint is a convention rather than a discovered quantity.
- The scan is finite because the score only changes when a row crosses the cutoff.
Where this goes next
This question is one step of Trees, and Committees of Them, a course that works through its subject from the beginning. The page that teaches this part of it is Growing a Whole Tree, and it reads the whole way through whether or not you have an account.
What an account changes is the practice: problems standing under every idea, marked with a reason rather than a tick, in a copy of the course that is yours.
Read that pageThe whole courseOther questions people ask about this
- Why do tree ensembles still beat neural networks on ordinary spreadsheet data?Because each spreadsheet column means something on its own, which is exactly what a one-column-at-a-time split needs; where inputs only mean something in combination, networks win by tens of points.
- What is Gini impurity actually measuring?Gini impurity is the chance that two rows picked independently from a group have different outcomes. It is zero for a pure group and largest for an even split.
- Why does changing a few rows of the training data give a completely different tree?A tree picks the single best split at each step, so a tiny data change can swap a near-tied split and reshape the whole tree; that is normal, not a bug.
- Why does a tree that gets every training example right do worse on new data?A tree that fits every training row has memorised individual cases rather than learned a pattern, so its accuracy on new rows falls.
- Why does the model get stuck repeating the same sentence over and over?The model is not stuck in a memory loop: once it has written a phrase, that phrase is in its context and makes writing it again more likely, so a decoder that always picks the top word cannot leave.
- Why does diffusion add Gaussian noise instead of any random corruption?Diffusion adds Gaussian noise because Gaussian errors combine exactly: each step keeps the spread fixed and gives a closed-form noisy picture at any later step, which makes the reverse step learnable.