You Think You Are Talking to a Server, and You Are Talking to Four Machines That Have Agreed Not to Mention It

Last timeNot Asking Again

A request passes through a proxy, a balancer and an edge before anything you wrote runs. Each one helps, each one changes what failure looks like, and each one hides something.

What is actually in the path

Everything so far has described two parties. There are rarely two. Between the

browser and the code you wrote there are usually three or four machines, none

of which appear anywhere in your source, and each of which can change the

answer.

FIG 1The real path of one request

Eight boxes, and your code is one of them. Two of the machines before it can answer without asking anybody, and two of them replace the address the request appears to come from. When a request behaves strangely, the first question is which of these boxes it got to.

FIG 2What each machine buys and what it costs

	can answer without reach	removes a failing copy f	reduces the distance tra	replaces or rewrites the
a reader proxy	1	0	0	1
a load balancer	0	1	1	1
a content network edge	1	1	0	1
a service mesh sidecar	1	0	1	0

The first marked cell is the one that breaks logging: by the time a request arrives, the address it came from is the address of the last machine, so every reader appears to be in one place. The second marked cell is the one worth paying for, because it converts a machine failure into a non-event. Every row costs something in the last column, which is why the forwarding header exists.

Choosing which copy serves you

A balancer does one thing well. It holds a list of copies of the application

and a health check for each, and it sends each new connection to a copy that is

currently passing.

The list is the easy part. The health check is the whole game, and getting it

wrong produces two opposite failures that are both worse than having no

balancer at all.

A check that is too shallow passes a copy that is broken. If the check asks

only whether the process is listening, a copy whose database connection pool is

exhausted answers the check instantly and fails every real request. Traffic

continues to be sent to it, and the service is a third broken with every

machine reporting healthy.

A check that is too deep fails every copy at once. If the check asks the copy

to query the database, then a slow database fails the check on all copies

simultaneously, the balancer removes all of them, and a degraded database

becomes a complete outage. This is the more common of the two mistakes and the

more dramatic.

The rule that survives contact with reality is that a health check should test

whether this copy can serve requests, and nothing about whether the things it

depends on are healthy, because those are shared and failing them together is

exactly what must not happen.

FIG 3One failing copy, four seconds

step	moment	what the copy did	what the balancer did	what readers saw	what happened
1	0s	serving normally	sending it a third of traffic	nothing unusual	Three copies, each taking a third. The steady state, and the only state most people ever picture.
2	1s	stopped answering	still sending it a third	a third of requests hanging	The gap between failing and being noticed. Whatever the health check interval is, this is how long it lasts, and it is the number to argue about.
3	3s	still down	removed it after two failed checks	nothing unusual	Two copies now carry everything. The remaining copies must have the headroom for this, which is the real reason capacity is planned above the average.
4	4s	restarted and passing checks	returning it to rotation gradually	nothing unusual	Gradually matters. A cold copy given a third of traffic instantly has an empty cache and no warm connections, and can fail again under the load it was just handed.

4 steps

The outage lasted two seconds and affected a third of requests, and the only reason it was that short is the check interval. Halving the interval halves the outage and doubles the chance of removing a healthy copy over a momentary blip, which is the trade the number encodes.

Answering from nearby

The second lesson established that distance costs round trips and that round

trips cannot be made faster. There is exactly one remedy, which is to be

closer, and a content network is that remedy industrialised.

The lesson stops here

2 more paragraphs to go

You have read the opening. The rest of the argument, the problems that check whether it landed, and the lines worth keeping at the end all come with a plan.

The first lesson of every course in the library reads the whole way through, free, so you can see exactly what the rest of them are.

See the plan The contents

This is the reading half

Starting the course gives you your own copy of it. Every idea on every page has problems standing under it, marked with a reason rather than a tick, and any sentence you do not believe can be opened and argued with. None of that can happen on a page nobody owns.

The contents