What Deserves an Alarm
Last timeThe Model Changing Its Own Inputs
An alarm nobody can act on is noise with a pager attached, so every threshold has to be computed from the ordinary variation and tied to something a person can actually do.
An alarm is a request for an action
The previous lessons produced a long list of things that can be measured. This
lesson is about the much shorter list of things worth waking somebody for, and
the distinction is not about how important the signal is. It is about whether
anybody can do anything.
Suppose the input drift number on a column doubles. Somebody is paged at two in
the morning. What do they do? They cannot retrain a model at two in the
morning, and they should not. They look at a chart, note that a column moved,
and go back to bed. The alarm was correct and useless, and its real effect was
to make the next alarm slightly less likely to be taken seriously.
| can somebody act within | is the action obvious | page | ticket | weekly report | |
|---|---|---|---|---|---|
| a required input is sudd | 1 | 1 | 1 | 0 | 0 |
| scoring requests are fai | 1 | 1 | 1 | 0 | 0 |
| volume has collapsed | 1 | 1 | 1 | 0 | 0 |
| a feature lookup is retu | 1 | 1 | 1 | 0 | 0 |
| an input column has drif | 0 | 0 | 0 | 1 | 1 |
| labelled accuracy has fa | 0 | 0 | 0 | 1 | 1 |
| the model is refusing mo | 0 | 0 | 0 | 0 | 1 |
The arithmetic of false alarms
The reason to be strict about this is that false alarms are not a nuisance,
they are the mechanism by which monitoring dies. And their rate is computable
before anything is built.
- how many checks run in a day
- the chance that one check fires when nothing is wrong
- days between false alarms
The same arithmetic in the other direction gives the threshold a set of checks
can afford. If sixty checks run hourly and one false alarm a week is the
budget, each check has to be set so it fires about once in ten thousand runs
when nothing is wrong, which is a much stricter threshold than anybody chooses
by eye.
The lesson stops here
4 more paragraphs to go
You have read the opening. The rest of the argument, the problems that check whether it landed, and the lines worth keeping at the end all come with a plan.
The first lesson of every course in the library reads the whole way through, free, so you can see exactly what the rest of them are.
See the planThe contentsThis is the reading half
Starting the course gives you your own copy of it. Every idea on every page has problems standing under it, marked with a reason rather than a tick, and any sentence you do not believe can be opened and argued with. None of that can happen on a page nobody owns.
The contents