The Model Changing Its Own Inputs

Last timeLabels That Arrive Weeks Later

A model whose decisions determine which cases ever get an outcome is measuring a world it created, and the resulting numbers improve while the decisions get worse.

The loop

Every lesson so far has treated the model as an observer of a world that moves

on its own. This lesson is about the case where the model is one of the things

moving the world, which is more common than it sounds.

A credit model decides who gets a loan, so only approved applicants ever

produce a repayment record. A fraud model blocks transactions, so blocked

transactions never produce a dispute or a lack of one. A recommender decides

what appears on the page, so only shown items can be clicked. In each case the

data available tomorrow is a consequence of the decisions made today.

FIG 1Where the loop is, and what it corrupts

	the model decides what d	the outcome of a refused	the next model trains on	measured quality drifts
credit approval	1	1	1	1
fraud blocking	1	1	1	1
recommendations	1	1	1	1
weather forecasting	0	0	1	0
image tagging	0	0	1	0

Coded 1 for yes and 0 for no. The last two rows are the comfortable case: the model observes and does not intervene, so the data keeps arriving regardless of what the model said. The first three are the ordinary commercial case, and in all three the final column is the dangerous property: the number on the dashboard gets better as the system gets worse.

Outcomes only for the approved

Take the credit case, which is the clearest.

A model approves seventy per cent of applicants. Those seventy per cent

eventually repay or do not, and three per cent of them default. That three per

cent is the measured bad rate, and it is a statement about approved applicants

only.

The thirty per cent who were refused have no outcome, ever. They did not take

a loan, so they did not default, so nothing is recorded. Whether the model was

right to refuse them is permanently unknown, and the only way it would become

known is if somebody had lent them money.

FIG 2What the refused cases must have looked like

: the share of applicants approved
: the bad rate among approved applicants, which is measured
: the bad rate of the whole applicant pool, which is not
: the bad rate the refused applicants would have had

Seventy per cent approved at a three per cent bad rate, against a whole pool rate of eight per cent, implies the refused group would have gone bad at about twenty per cent, which is a good refusal. The problem is that the whole pool rate on the right is exactly the quantity nobody has, so this expression is a way of stating an assumption clearly rather than a way of computing anything. Every method for the refused cases is an assumption of this kind, and the honest ones say so.

The consequence is a drift with a comfortable direction. Tighten the approvals

and the measured bad rate falls, because the marginal cases were the risky

ones. The next model is trained on a cleaner population, learns a sharper rule

for that population, and refuses slightly more. Every number improves. The

business is turning away a growing number of people who would have repaid, and

nothing in the data says so.

The lesson stops here

5 more paragraphs to go

You have read the opening. The rest of the argument, the problems that check whether it landed, and the lines worth keeping at the end all come with a plan.

The first lesson of every course in the library reads the whole way through, free, so you can see exactly what the rest of them are.

See the plan The contents

This is the reading half

Starting the course gives you your own copy of it. Every idea on every page has problems standing under it, marked with a reason rather than a tick, and any sentence you do not believe can be opened and argued with. None of that can happen on a page nobody owns.

The contents

The Model Changing Its Own Inputs

The loop

Outcomes only for the approved

5 more paragraphs to go

This is the reading half

Read alongside