A Pager That Everyone Trusts

Last timeHow a Small Problem Becomes a Large One

An alert is a claim that a person should stop what they are doing right now. Most alerts cannot support that claim, and the ones that can are ruined by the ones that cannot.

An alert is an interruption with a claim attached: this matters more than

whatever you were doing. Most alerts in most systems cannot support that claim,

and the damage is not the interruption itself.

FIG 1Deciding what an alert should do

Almost every overgrown alerting system is full of conditions that fail one of these two questions and were added because somebody was once surprised. Running the list through this diagram usually removes most of it.

Alerting on the budget instead of a threshold

A fixed threshold has an impossible job: tight enough to catch a real problem

quickly, loose enough to ignore a blip. Asking how fast the budget is being

consumed solves this, because a brief blip spends very little of it.

FIG 2How long the monthly allowance lasts at a given burn rate

hours remaining

At the sustainable rate the allowance lasts the whole month, which is what sustainable means. At ten times it is gone in three days, and at thirty times in a single day. The curve is steep at the left, which is why the interesting thresholds sit between two and fifteen.

FIG 3The same measurement, three different responses

	burn rate	window watched, in hours	hours until the allowanc
page immediately	14.4	1.0	2.0
open a ticket	6.0	6.0	12.0
note it and move on	1.0	72.0	720.0

One signal produces three different urgencies depending on how fast it is moving. The fast row uses a short window so it responds quickly, and the slow rows use long windows so that a brief incident does not trigger them. That pairing of rate with window is the whole recipe.

FIG 4Pages per shift

Each page costs more than the minutes it takes: an investigation, a write-up, and a disturbed night that degrades the next day. Past the ceiling the behaviour changes qualitatively, and the alert that finally matters arrives into a system that has been trained to dismiss it.

The work that grows

Toil is operational work that is manual, repetitive, and grows in proportion to

the size of the service. Restarting something by hand. Running a migration

step by step. Approving a capacity request that nobody reviews. Each instance is

small, which is why it is tolerated, and each one gets more frequent as the

system grows.

Capping it is the standard practice: a fixed share of the team's time, often

half, above which the team stops taking on new operational surface and works on

removing what it already has. The cap matters because toil crowds out exactly

the work that would reduce it, so without one the share only moves in one

direction.

The lesson stops here

2 more paragraphs to go

You have read the opening. The rest of the argument, the problems that check whether it landed, and the lines worth keeping at the end all come with a plan.

The first lesson of every course in the library reads the whole way through, free, so you can see exactly what the rest of them are.

See the plan The contents

This is the reading half

Starting the course gives you your own copy of it. Every idea on every page has problems standing under it, marked with a reason rather than a tick, and any sentence you do not believe can be opened and argued with. None of that can happen on a page nobody owns.

The contents

A Pager That Everyone Trusts

Alerting on the budget instead of a threshold

The work that grows

2 more paragraphs to go

This is the reading half

Read alongside