The Helpful Client That Finishes the Job

Last timeMost Outages Are Deployments

A retry is a reasonable response to one failed request and a catastrophic response to a failing service, because it adds load exactly when there is none to spare.

A retry is obviously correct. The request failed, the failure was probably

transient, so try again. Every part of that reasoning is sound for one request

and wrong for a million.

FIG 1What a retry policy does to the offered load

: the requests the users actually made
: the original attempt plus however many retries are permitted

The multiplier applies only when things are failing, which is the worst possible time for it. A service running at ninety percent of capacity and beginning to fail does not need three times the traffic, and three times the traffic is exactly what a well-intentioned client library delivers.

Layers

The multiplier is bad. What makes it dangerous is that it composes.

FIG 2One user action, four layers, three attempts each

	layer	attempts per call	requests reaching this l
browser to gateway	1	3	3
gateway to service	2	3	9
service to backend	3	3	27
backend to database	4	3	81

Eighty-one database queries for one click. No one wrote this down and agreed to it. Each layer made a locally reasonable choice and the product was never examined, which is the usual shape of this failure and the reason retry policy should live at exactly one layer.

FIG 3Why removing the original cause does not help

The only ways out are to cut the load, by shedding or by making the clients stop, or to add capacity faster than the loop can consume it, which is rarely possible at the speed required.

Reject early

When the work cannot be done, the cheapest thing to do is say so immediately.

The lesson stops here

3 more paragraphs to go

You have read the opening. The rest of the argument, the problems that check whether it landed, and the lines worth keeping at the end all come with a plan.

The first lesson of every course in the library reads the whole way through, free, so you can see exactly what the rest of them are.

See the plan The contents

This is the reading half

Starting the course gives you your own copy of it. Every idea on every page has problems standing under it, marked with a reason rather than a tick, and any sentence you do not believe can be opened and argued with. None of that can happen on a page nobody owns.

The contents

The Helpful Client That Finishes the Job

Layers

Reject early

3 more paragraphs to go

This is the reading half

Read alongside