Running a Service That Stays Up
Nothing Degrades Gracefully
Last timeWhat Dependencies Cost
Waiting time does not rise in step with load. It rises with one over the remaining headroom, which means the last ten percent of capacity costs more than the first ninety.
Load rises smoothly. Latency does not. That mismatch is responsible for a large
share of outages that arrive with no warning, and it has a shape worth knowing
by heart.
Before any of it is arithmetic it is an ordinary scene. One clerk serves one
person at a time, and whoever the clerk cannot take right now waits in a line
behind a rope. Nothing about that line looks interesting while there is empty
rope left. What the rest of this lesson does is say precisely what happens as
the empty rope runs out.
The curve
The denominator is the whole story. What governs the wait is not how much
capacity is being used but how much is left, and as that remainder approaches
zero its reciprocal approaches infinity.
| step | fraction of capacity in use | time in the system | increase over the previous step | what happened |
|---|---|---|---|---|
| 1 | 0.5 | 2 | 1 | Half the capacity in use. Each job takes twice its service time, which nobody notices. |
| 2 | 0.8 | 5 | 2.5 | A large traffic increase, and the response time has only gone up by a factor of two and a half. |
| 3 | 0.9 | 10 | 2 | A much smaller traffic increase, and the response time doubles again. |
| 4 | 0.95 | 20 | 2 | A tiny traffic increase, and it doubles once more. This is the step people describe as the service falling over. |
- how many jobs are in the system at any moment
- how many arrive per second
- how long each one spends in the system
The relation that never lies
Used in the other direction it also sets a bound: if a thousand requests arrive
per second and each must finish within a tenth of a second, then no more than a
hundred can be in flight at once. That is a concurrency limit derived rather
than guessed, and it is the right number to enforce.
The lesson stops here
2 more paragraphs to go
You have read the opening. The rest of the argument, the problems that check whether it landed, and the lines worth keeping at the end all come with a plan.
The first lesson of every course in the library reads the whole way through, free, so you can see exactly what the rest of them are.
See the planThe contentsThis is the reading half
Starting the course gives you your own copy of it. Every idea on every page has problems standing under it, marked with a reason rather than a tick, and any sentence you do not believe can be opened and argued with. None of that can happen on a page nobody owns.
The contents