The Service Was Fine Until Somebody Changed It

Last timeWhy It Falls Over All at Once

Steady systems do not spontaneously break. Most incidents begin with a change, which makes how changes are released the most productive place to spend reliability effort.

Leave a service completely alone and it will mostly keep running. That is an

uncomfortable observation, because it means the main source of unreliability is

the same activity that makes the service worth having.

Expected damage

Releasing is a gamble with three terms, and only one of them is easy to change.

FIG 1What a release is expected to cost

: the chance this release is bad
: the share of traffic exposed to it
: how long it runs before it is caught and undone

Reducing the chance of a bad release means better tests and more review, which is slow and has diminishing returns. Reducing the exposure costs nothing but arrangement. That asymmetry is why staged rollout is the first thing to build and better testing is the second.

FIG 2A rollout with somewhere to stop

What makes this work is not the staging but the gates. A rollout that advances on a timer regardless of what the small stage showed has bought exposure reduction for the first few minutes and nothing after that.

FIG 3The same bad release, released three ways

	percent of traffic expos	minutes before rollback	user-minutes lost, relat
straight to everyone	100.00	45.00	45.00
staged, checked by a per	10.00	12.00	1.20
staged, checked automati	1.00	6.00	0.06

Three orders of magnitude between the top and bottom rows, with the same code and the same defect. The last column is the product of the first two, which is the formula above doing its work. Note that the automation improved both terms at once.

The canary only counts if something reads it

The common failure is to build the staged rollout and then compare the stages by

eye. That leaves the slowest component of the loop in place. The comparison

wants to be an automatic one against the unchanged traffic, on a small number of

signals: error rate, latency at the tail, and one or two things specific to the

service. If those differ beyond a threshold, the rollout reverses itself without

asking.

The lesson stops here

3 more paragraphs to go

You have read the opening. The rest of the argument, the problems that check whether it landed, and the lines worth keeping at the end all come with a plan.

The first lesson of every course in the library reads the whole way through, free, so you can see exactly what the rest of them are.

See the plan The contents

This is the reading half

Starting the course gives you your own copy of it. Every idea on every page has problems standing under it, marked with a reason rather than a tick, and any sentence you do not believe can be opened and argued with. None of that can happen on a page nobody owns.

The contents

The Service Was Fine Until Somebody Changed It

Expected damage

The canary only counts if something reads it

3 more paragraphs to go

This is the reading half

Read alongside