Each Tree Fixing the Last

Last timeAveraging Many Trees

Instead of averaging independent trees, each new tree is fitted to what the committee so far got wrong, which is more accurate and far less forgiving.

A different way to combine

A forest grows every tree independently and averages. Each tree does the whole

job badly and the average does it well.

This lesson does the opposite. Trees are grown one after another, and each one

is told what the committee so far got wrong and asked to fix only that. No

tree does the whole job; the sum of them does.

FIG 1Four trees in sequence, predicting a house price

step	after this tree	what it was fitted to	its contribution	prediction	still wrong by
1	starting guess	nothing	the average, 420	420	-115
2	tree 1	the -115 and its neighbours	-61	359	-54
3	tree 2	the -54 and its neighbours	-29	330	-25
4	tree 3	the -25 and its neighbours	-13	317	-12
5	tree 4	the -12 and its neighbours	-7	310	-5

5 steps

Prices in thousands; the true price is 305. Each tree is fitted to the remaining error of every row, not just this one, so its contribution here is what it predicts for rows like this one. The remaining error shrinks by roughly half each time, which is the multiplier at work, and the sequence closes on the truth from one side rather than jumping around it.

The loop

Start with a single number for everybody, the average outcome. Compute how

wrong that is for each row. Fit a shallow tree to those errors. Add a fraction

of its predictions to the running total. Compute the new errors. Repeat.

FIG 2One round

: the committee after m trees
: the tree fitted at round m
: the multiplier, typically between 0.01 and 0.1
: how wrong the committee is on row i at the start of this round

The target of each tree is the leftover r rather than the outcome y. For a squared score the leftover is exactly the slope of the score with respect to the prediction, with the sign flipped, which is why this is the same descent as the third lesson of the previous course with trees in place of knob adjustments.

That connection is worth holding onto. The previous course took steps downhill

by adjusting numbers. This takes steps downhill by adding functions. The step

size is the same idea wearing the same name, and the reason for keeping it

small is the same too.

The lesson stops here

5 more paragraphs to go

You have read the opening. The rest of the argument, the problems that check whether it landed, and the lines worth keeping at the end all come with a plan.

The first lesson of every course in the library reads the whole way through, free, so you can see exactly what the rest of them are.

See the plan The contents

This is the reading half

Starting the course gives you your own copy of it. Every idea on every page has problems standing under it, marked with a reason rather than a tick, and any sentence you do not believe can be opened and argued with. None of that can happen on a page nobody owns.

The contents

Each Tree Fixing the Last

A different way to combine

The loop

5 more paragraphs to go

This is the reading half

Read alongside