Basis Lab

Putting a model into service

Know whether a model that is already serving people is still working, by measuring what you can measure without labels, detecting the ways the world moves away from the training data, and deciding when a drop in a number justifies retraining

Watching a Model After Launch

A model is at its best on the day it ships. This course is about the months after, when the labels are gone and the world keeps moving.

8 lessons, written and corrected before you arrived. Reading them here needs no account. The first reads the whole way through; the others open and then stop, because a page nobody owns cannot tell who is reading it. Starting the course gives you your own copy, where every idea has problems standing under it and you can ask about any sentence.

Start reading

This is course 9 of 9 in a reading track: put a model in front of people and know whether it is working. The track is an order, not a gate, and the rest of it is on one page.

  1. 01What You Can Measure Without LabelsOn the day after launch the true answers stop arriving, so the instrument panel is built from inputs, scores and what people do next rather than from accuracy.
  2. 02When the Inputs Moveopening onlyComparing this week of incoming data against the training data is easy; the hard part is knowing how much difference an ordinary week produces, so that noise does not become an alarm.
  3. 03When the Right Answer Movesopening onlyThe inputs moving is visible and usually harmless; the relationship between input and answer moving is invisible without labels and is what silently ruins a model.
  4. 04Labels That Arrive Weeks Lateropening onlyWhen the truth is known only weeks afterwards, the cases that reported back first are the unrepresentative ones, and judging a model on them is worse than waiting.
  5. 05The Model Changing Its Own Inputsopening onlyA model whose decisions determine which cases ever get an outcome is measuring a world it created, and the resulting numbers improve while the decisions get worse.
  6. 06What Deserves an Alarmopening onlyAn alarm nobody can act on is noise with a pager attached, so every threshold has to be computed from the ordinary variation and tied to something a person can actually do.
  7. 07Deciding to Retrainopening onlyRetraining is neither free nor safe, so the decision is a comparison between the measured decay and the cost, and a model retrained on a schedule nobody questions is its own failure mode.
  8. 08The Failures That Make No Noiseopening onlyA broken input, a stale lookup and a confident answer about something never seen are the three production failures that leave every quality number looking normal.