Answers · How models work
What is Gini impurity actually measuring?
In shortGini impurity is the chance that two rows picked independently from a group have different outcomes. It is zero for a pure group and largest for an even split.
Gini impurity is the chance that two rows drawn independently from a group have different outcomes. If a fraction \(p\) of the group has the first outcome, then \(G=1-p^2-(1-p)^2=2p(1-p)\), which is zero for a pure group and \(1/2\) for an even split.
Why the formula is a probability
The formula is not a convention. Draw one row: the chance it has the first outcome is \(p\), and the chance a second independent row also has it is \(p^2\). The chance both have the other outcome is \((1-p)^2\). Those are the two ways two draws can agree, so the chance they disagree is one minus their sum. Expanding \(1-p^2-(1-p)^2\) gives \(2p(1-p)\). The maximum for a binary group is \(1/2\), at \(p=1/2\), because that is the most disagreement two draws can have. A group that is 90 per cent one outcome has impurity \(2\cdot0.9\cdot0.1=0.18\), less than half but not zero. That is the sense in which lower is better: it is a smaller chance of disagreement, not a smaller arbitrary penalty.
The arithmetic on twelve applications
Take twelve loan applications, seven repaid and five not. The fraction repaid is \(p=7/12\), and the impurity before any question is the chance two random applications disagree. The calculation is short because the two squares collapse to a product.
Now ask whether income is above thirty thousand. The yes group has seven rows, six repaid; the no group has five rows, one repaid. Each side has its own impurity, computed the same way, and the split is scored by the weighted average of the two sides, weighted by seven twelfths and five twelfths. The yes group is mostly repaid but not pure, so its impurity is \(12/49\), about 0.245. The no group is mostly not repaid, with impurity \(8/25=0.32\). Neither side is clean, but both are less mixed than the original 0.486, and the weighted average 0.276 says so.
The weighting is not decoration. If the yes side has seven rows and the no side five, the after-split mixedness is the average a row would experience: seven twelfths of the time you are in the yes group with its impurity, five twelfths in the no group with its impurity. Without weights, a question that peels off a single pure row would look perfect, because one side would be pure and the other eleven mixed rows would be ignored. With weights, that one row contributes one twelfth of the average and the eleven mixed rows carry the rest.
The value of the question is the impurity before minus the weighted impurity after. For income above thirty, that is \(35/72 - 29/105 = 529/2520 \approx 0.210\). The units are probability: the question lowers the chance that two rows in the same child disagree by about 0.210. A value near zero would mean the question barely changes that chance. The best question is the one with the largest reduction, which is why lower child impurity is better.
What the number is not
The natural guess is that impurity is the error rate you get by labelling every row in a group with the group's majority outcome. It is not. In the yes group above the majority is repaid, and labelling all seven rows repaid would be wrong for the one row not repaid, an error rate of \(1/7\approx0.143\). The Gini impurity of that same group is \(12/49\approx0.245\). The two numbers agree at an even split and at the pure ends, but between them they answer different questions. Majority error asks how often one fixed label fails. Gini impurity asks how often two independent draws from the group disagree.
Where the reading changes
For more than two outcomes the formula becomes \(G=1-\sum_k p_k^2\), summing over every outcome. The disagreement reading survives: draw two rows, and \(G\) is still the chance their outcomes differ. The binary version above is the case with two terms. If you change the labelling rule, for instance from drawing a label from the group's proportions to always using the majority, you are measuring something else, and the binary number that answers that question is \(\min(p,1-p)\), not Gini impurity. Choosing a split is therefore a search over questions, scored by how much each one reduces this probability. The search is finite for a numeric column because every threshold between the same pair of neighbouring sorted values produces the same two groups, so only the gaps need trying. The score is not an arbitrary index; it is a difference of two probabilities of disagreement.
Worth keeping
- For a binary group with fraction p in one class, Gini impurity is 2p(1-p).
- It is not the error rate of the majority label; it is the probability that two independent draws disagree.
- A split is scored by the impurity before it minus the weighted average impurity after it.
Where this goes next
This question is one step of Trees, and Committees of Them, a course that works through its subject from the beginning. The page that teaches this part of it is Splitting on One Question, and it reads the whole way through whether or not you have an account.
What an account changes is the practice: problems standing under every idea, marked with a reason rather than a tick, in a copy of the course that is yours.
Read that pageThe whole courseOther questions people ask about this
- Why do tree ensembles still beat neural networks on ordinary spreadsheet data?Because each spreadsheet column means something on its own, which is exactly what a one-column-at-a-time split needs; where inputs only mean something in combination, networks win by tens of points.
- How does a decision tree pick the exact number to split a column on?A decision tree sorts the column, tries a cutoff between every adjacent pair of distinct values, scores each one by how mixed the two resulting groups are, and keeps the best.
- Why does changing a few rows of the training data give a completely different tree?A tree picks the single best split at each step, so a tiny data change can swap a near-tied split and reshape the whole tree; that is normal, not a bug.
- Why does a tree that gets every training example right do worse on new data?A tree that fits every training row has memorised individual cases rather than learned a pattern, so its accuracy on new rows falls.
- Why does the model get stuck repeating the same sentence over and over?The model is not stuck in a memory loop: once it has written a phrase, that phrase is in its context and makes writing it again more likely, so a decoder that always picks the top word cannot leave.
- Why does diffusion add Gaussian noise instead of any random corruption?Diffusion adds Gaussian noise because Gaussian errors combine exactly: each step keeps the spread fixed and gives a closed-form noisy picture at any later step, which makes the reverse step learnable.