Labels That Arrive Weeks Later

Last timeWhen the Right Answer Moves

When the truth is known only weeks afterwards, the cases that reported back first are the unrepresentative ones, and judging a model on them is worse than waiting.

The newest weeks cannot be judged

A model decides today. The outcome is known in six weeks, or six months, or

not at all. This is not an edge case: credit, fraud, churn, medical follow up,

hiring and insurance all work this way, which is most of the places a model is

making consequential decisions.

The first consequence is arithmetic. If the label takes forty five days and it

takes another three weeks of volume to accumulate enough labelled cases to

measure anything, then the quality figure available today describes the model

as it was two months ago.

FIG 1How old the measurable quality figure is

: the delay before an outcome is known, in days
: decisions made per day
: labelled cases needed to measure what you want to measure
: how far behind the present the quality figure is

Forty five days of delay, nine hundred decisions a day and twenty thousand cases needed gives sixty seven days, so the newest reliable accuracy figure describes decisions made more than two months ago. Nothing fixes this, and the useful response is to accept it, state the lag explicitly on the dashboard, and treat the unlabelled signals from the previous lessons as the instrument for the recent period.

The cases that report back first

The tempting repair is to read the labels that have arrived so far. This is

worse than waiting, and the reason is worth being precise about.

Outcomes do not arrive in a random order. The fastest ones are the extreme

ones. A fraudulent transaction is disputed within days, while a legitimate one

is never disputed at all. A loan that goes wrong immediately misses its first

payment within a month, while one that repays takes the full term to say so.

So the early arrivals are not a sample of the cohort, they are the bad tail of

it.

FIG 2What the late reporters must look like

: the share of the cohort that has reported so far
: the bad rate among those early reporters
: the bad rate of the whole cohort, known from history
: the bad rate the remaining cases must have

Thirty per cent reported so far with an eighteen per cent bad rate, against a historical whole cohort rate of nine per cent, implies the remaining seventy per cent will come in at about five per cent. The arithmetic is worth doing explicitly because it converts an alarming partial read into a statement about what has to happen next, and in this case nothing alarming has happened at all.

FIG 3One cohort, read at five moments

step	days since the decisions	share of labels arrived	bad rate among arrived	what a naive read concludes
1	10	8 per cent	41 per cent	catastrophe, escalate immediately
2	30	26 per cent	19 per cent	much worse than the last model
3	60	58 per cent	12 per cent	somewhat worse
4	90	87 per cent	9.4 per cent	about the same
5	120	97 per cent	8.9 per cent	the truth: slightly better

5 steps

The same cohort throughout, with nothing changing except how much of it has reported. A team reading at day ten sees a bad rate more than four times the truth and will roll the model back. The pattern is so reliable that the shape of this curve for a given product is itself worth recording, since it is what turns a partial read into a usable one.

Waiting for a cohort to mature

The clean fix is to stop reporting by calendar week and start reporting by

cohort.

Group the cases by when the decision was made. Read a cohort only once its

labels are substantially complete, which for most products means a fixed

number of days chosen from the arrival curve, typically where ninety per cent

have reported. Every cohort is then read at the same point in its life, so the

comparison between cohorts is honest.

The lesson stops here

2 more paragraphs to go

You have read the opening. The rest of the argument, the problems that check whether it landed, and the lines worth keeping at the end all come with a plan.

The first lesson of every course in the library reads the whole way through, free, so you can see exactly what the rest of them are.

See the plan The contents

This is the reading half

Starting the course gives you your own copy of it. Every idea on every page has problems standing under it, marked with a reason rather than a tick, and any sentence you do not believe can be opened and argued with. None of that can happen on a page nobody owns.

The contents