Treat reliability as a measured quantity with a budget attached, well enough to set a target, defend it against the arithmetic of dependencies and queues, and tell which failures are worth paging a person about

Running a Service That Stays Up

Reliability is a number you choose and then spend. This course covers how to measure it, how dependencies and queues destroy it, and how change, overload and alerting are managed against a budget.

8 lessons, written and corrected before you arrived. Reading them here needs no account. The first reads the whole way through; the others open and then stop. Starting the course gives you your own copy, where every idea has problems standing under it and you can ask about any sentence.

Start reading

01Available to Whom, ExactlyAvailability is a ratio of good events to total events, and every word in that sentence has to be pinned down before the number means anything at all.
02The Gap Between Your Target and Perfection Is a Resourceopening onlySubtract the target from one and you get a budget: an amount of failure you are allowed to produce. Spending it deliberately is the whole method.
03The Average Is the Enemyopening onlyA mean response time can look excellent while a large minority of users are having a terrible experience, and a page that makes many calls meets the slow tail on almost every load.
04A Chain of Reliable Parts Is Not Reliableopening onlyAvailabilities multiply along a chain, so a service standing on ten dependencies cannot be more reliable than their product, however good its own code is.
05Nothing Degrades Gracefullyopening onlyWaiting time does not rise in step with load. It rises with one over the remaining headroom, which means the last ten percent of capacity costs more than the first ninety.
06The Service Was Fine Until Somebody Changed Itopening onlySteady systems do not spontaneously break. Most incidents begin with a change, which makes how changes are released the most productive place to spend reliability effort.
07The Helpful Client That Finishes the Jobopening onlyA retry is a reasonable response to one failed request and a catastrophic response to a failing service, because it adds load exactly when there is none to spare.
08A Pager That Everyone Trustsopening onlyAn alert is a claim that a person should stop what they are doing right now. Most alerts cannot support that claim, and the ones that can are ruined by the ones that cannot.