Answers · How models work
Why do tree ensembles still beat neural networks on ordinary spreadsheet data?
In shortBecause each spreadsheet column means something on its own, which is exactly what a one-column-at-a-time split needs; where inputs only mean something in combination, networks win by tens of points.
Because a spreadsheet column was put there by somebody who chose it and named it, so it carries meaning on its own, and a tree does nothing except ask one column at a time whether it clears a threshold. That is the right question for that shape of data. When the inputs mean something only in combination, as pixels, audio samples and words do, the same one-column-at-a-time restriction becomes fatal, and networks win there by tens of accuracy points rather than by one or two.
Take a table with twenty columns: age, income, postcode band, number of previous orders, and sixteen more. One split is one column against one threshold.
A path of depth five asks five such questions, so it reads five of the twenty columns and ignores the other fifteen for the rows that take that path. A rule saying income above 30,000, age under 40, more than three orders in the last year is a legible sentence about a customer, and sentences of that form are frequently the thing that actually predicts the outcome. The fitting procedure searches the thresholds and keeps the one that leaves the two sides most uniform, which is zero work beyond sorting a column.
The preparation advantage follows from the same fact and is worth spelling out, because it is where most of the elapsed time goes. A split at income above 30,000 and a split at log income above 10.309 are the same split: , and any monotone transform carries thresholds to thresholds without changing which rows fall left and which fall right. Standardising a column, logging it, or clipping its extremes therefore leaves the set of available splits untouched, so the afternoon a network needs spent on scaling buys a forest exactly nothing. Categories are handled by asking whether a row's category is in a set, not by expanding twenty levels into twenty columns, and a missing value can be routed down a default branch fixed during fitting.
Why one column at a time stops working
A 28 by 28 greyscale digit is 784 numbers. The value in position 412 means nothing by itself: it means something only next to 411 and 413, and the same digit shifted twenty pixels to the left is a wholly different vector of 784 values. A depth-ten path through a tree asks ten questions about ten of those 784 columns. A fully grown tree of that depth has 1,024 leaves, so it has carved the input space into 1,024 axis-aligned boxes, and the set of images that count as a three is not usefully a union of a thousand axis-aligned boxes. A network's first layer sees all 784 at once and builds from weighted combinations of neighbours, which is the operation the data demands and the one the tree cannot perform at any depth.
| what the inputs are | one input means something alone | which family wins | typical gap |
|---|---|---|---|
| spreadsheet columns, mixed types | yes | trees | 3 |
| sensor readings over time | yes | trees | 1 |
| pixels of an image | no | networks | 40 |
| audio samples | no | networks | 50 |
| words of a document | no | networks | 35 |
The natural guess is that this is a tuning problem: that a wider network, a better schedule and more epochs would close the gap, and that trees are a habit people have not yet dropped. The guess fails because the gap is not a statement about how well either model was fitted. It is a statement about which hypotheses each family can reach cheaply. On a table the network has to spend data learning that column seven matters independently of column eight, a fact the tree gets for free from its hypothesis class; and its very first operation is a weighted sum across all twenty columns, which adds a product code of 7 to an order count of 7 before anything has been decided about either. No amount of tuning removes those two handicaps, because neither is a defect of fitting. What tuning cannot fix, the data can: on tables whose columns genuinely interact, the ranking flips back.
The cost of guessing wrong is not symmetric either. Put a network on spreadsheet data and you give up around three accuracy points and spend days instead of minutes; put a forest on images and you give up around forty. Three points is a tuning problem, worth revisiting once a baseline exists. Forty is not a tuning problem, it is the wrong tool, and no schedule recovers it.
Where it stops being true
Every tree prediction is an average of training rows that landed in one leaf, so no prediction can fall outside the smallest and largest values the tree was fitted on. Feed it an input beyond that range and it reports the nearest edge leaf, and keeps reporting it however far out you go. If the truth is and the data stops at , the model tracks it to and then sits at 16.5. At the truth is against a prediction of 16.5, an error of 14.5, which is 47 per cent of the correct value. Inside the observed range the same effect is cosmetic, a smooth curve coming out as a staircase; outside it, the model is not uncertain, it is systematically flat. Anything with a trend in it, prices over years, usage that grows, a measurement that drifts, meets this immediately. The check is one line: compare each input's range in the new data against its range in the training data.
Two further boundaries are worth knowing before the choice is made. A table whose columns interact heavily, with no column carrying much signal alone, behaves like the pixel case and the forest loses its edge; the shape of the inputs decides, not the fact that the data arrived as rows. And both families produce models nobody can read: a single tree is a set of rules a person can follow, a committee of hundreds is not, so if the answer has to be justified to a customer or a regulator, that requirement belongs at the start of the project, when it is still cheap, rather than at the end.
Worth keeping
- A split compares one column against a threshold, so scale, skew and monotone transforms cannot change which splits are available.
- A committee of trees cannot predict outside the range of values it was fitted on, so compare new inputs against training ranges before shipping one.
- Choosing a network for spreadsheet data costs a few accuracy points and days of work; choosing trees for images costs tens of points.
Where this goes next
This question is one step of Trees, and Committees of Them, a course that works through its subject from the beginning. The page that teaches this part of it is When to Reach for a Tree, and it reads the whole way through whether or not you have an account.
What an account changes is the practice: problems standing under every idea, marked with a reason rather than a tick, in a copy of the course that is yours.
Read that pageThe whole courseOther questions people ask about this
- How does a decision tree pick the exact number to split a column on?A decision tree sorts the column, tries a cutoff between every adjacent pair of distinct values, scores each one by how mixed the two resulting groups are, and keeps the best.
- What is Gini impurity actually measuring?Gini impurity is the chance that two rows picked independently from a group have different outcomes. It is zero for a pure group and largest for an even split.
- Why does changing a few rows of the training data give a completely different tree?A tree picks the single best split at each step, so a tiny data change can swap a near-tied split and reshape the whole tree; that is normal, not a bug.
- Why does a tree that gets every training example right do worse on new data?A tree that fits every training row has memorised individual cases rather than learned a pattern, so its accuracy on new rows falls.
- Why does the model get stuck repeating the same sentence over and over?The model is not stuck in a memory loop: once it has written a phrase, that phrase is in its context and makes writing it again more likely, so a decoder that always picks the top word cannot leave.
- Why does diffusion add Gaussian noise instead of any random corruption?Diffusion adds Gaussian noise because Gaussian errors combine exactly: each step keeps the spread fixed and gives a closed-form noisy picture at any later step, which makes the reverse step learnable.