Running a Service That Stays Up
Available to Whom, Exactly
Availability is a ratio of good events to total events, and every word in that sentence has to be pinned down before the number means anything at all.
Ask three people whether a service is up and you will get three answers, all
sincere. The purpose of this lesson is to replace the question with one that has
a single answer.
The ratio
Availability is the share of events that went well.
- the number of events that counted as successful
- the total number of events considered
An event is usually a request, but not always. For a batch system it might be a
scheduled run. For a data pipeline it might be an hour of freshness. Pick the
thing the user would complain about if it went wrong, and count those.
Measured where
This is the part that gets skipped, and it is the part that decides whether the
number is worth anything.
The rule is simple to state and unpopular to follow: measure as close to the
user as you can reach, and treat the service's own opinion of itself as
diagnostic information rather than as the number.
What counts as failure
Two cases decide most of the argument.
A response that arrives after thirty seconds is a failure. The user has left. So
the definition of success needs a latency limit inside it, and that limit is a
product decision rather than an engineering one.
A request that was rejected because it asked for something it was not allowed to
have is not a failure. The service did exactly its job. Counting those makes the
number worse every time someone probes an endpoint they should not, which
punishes the service for working correctly.
| hours down | hours in the window | availability percent | |
|---|---|---|---|
| reported over a day | 1.00 | 24.00 | 95.83 |
| reported over a month | 1.00 | 720.00 | 99.86 |
| reported over a year | 1.00 | 8760.00 | 99.99 |
Nines as minutes
The last step is to stop saying nines and start saying minutes, because minutes
are something a team can reason about.
| target percent | minutes down per month | minutes down per day | |
|---|---|---|---|
| two nines | 99.00 | 432.00 | 14.40 |
| three nines | 99.90 | 43.20 | 1.44 |
| four nines | 99.99 | 4.32 | 0.14 |
Each additional nine costs roughly an order of magnitude more to reach, and buys
a reduction in downtime that the user may not be able to perceive at all,
particularly if their own network is less reliable than your service. Choosing a
target is therefore an economic question, and the next lesson is about what the
gap between the target and perfection is for.
What to hold on to
Availability is good events over total events, and both halves are definitions
you choose. It has to be measured where the user stands, because the worst
failures are invisible from inside. Slowness counts as failure and expected
rejections do not. Every number needs a window attached. And a target is best
read as a number of minutes, because that is the form in which it can be spent.
Recap
- Availability is the share of events that succeeded, so the number depends entirely on what counts as an event and what counts as success, and those definitions are decisions rather than facts.
- The only measurement that matters is the one taken where the user is, because a service can be perfectly healthy by its own account while nobody can reach it.
- A target expressed in nines is a permission slip written in minutes: three nines is about forty-three minutes of unavailability per month, and those minutes are a resource to be spent rather than a failure to be avoided.
This is the reading half
Starting the course gives you your own copy of it. Every idea on every page has problems standing under it, marked with a reason rather than a tick, and any sentence you do not believe can be opened and argued with. None of that can happen on a page nobody owns.
The contents