When the Inputs Move

Last timeWhat You Can Measure Without Labels

Comparing this week of incoming data against the training data is easy; the hard part is knowing how much difference an ordinary week produces, so that noise does not become an alarm.

The comparison itself is easy

Take a column, divide its range into ten buckets fixed once against the

training data, and count what share of this week falls into each. Compare

those shares against the training shares. Sum one term per bucket. One number

per column per week.

FIG 1One column, two weeks, bucket by bucket

	share in training, per c	share this week, per cen	difference, per cent
under 20	18	11	-7
20 to 30	24	19	-5
30 to 40	22	21	-1
40 to 50	17	20	3
50 to 65	12	19	7
over 65	7	10	3

Customer age, in a service whose audience has aged over a year of operation. Every bucket moved a little and the movement is in a consistent direction, which is the signature of a real shift rather than noise. Summing the sizes of the differences gives twenty six points of total movement, and the useful discipline is to record that one number weekly so that its own history says what normal looks like.

The arithmetic has several standard forms, and which one is used matters much

less than three practical rules. Fix the buckets once, against the reference,

or a widening distribution will look stable because the buckets widened with

it. Give every bucket a floor so that an empty bucket does not produce an

infinite or undefined term. And compare against the training data, not against

last week, because comparing against last week makes a slow steady drift

invisible: each week looks like the last one right up to the point where the

model is answering about a world it never saw.

Telling noise from a shift

The number moves every week whether anything happened or not, and the size of

that movement is computable, which means the threshold does not have to be

guessed.

The lesson stops here

5 more paragraphs to go

You have read the opening. The rest of the argument, the problems that check whether it landed, and the lines worth keeping at the end all come with a plan.

The first lesson of every course in the library reads the whole way through, free, so you can see exactly what the rest of them are.

See the plan The contents

This is the reading half

Starting the course gives you your own copy of it. Every idea on every page has problems standing under it, marked with a reason rather than a tick, and any sentence you do not believe can be opened and argued with. None of that can happen on a page nobody owns.

The contents

When the Inputs Move

The comparison itself is easy

Telling noise from a shift

5 more paragraphs to go

This is the reading half

Read alongside