A System Design in Forty Minutes
What a Reader Sees While Something Is Broken
Last timeWhere the Writes Go
A failure exposes a promise, so the promise is priced here in round trips, in machines and in the records a lagging copy cannot serve, once for the link row and once for the click counter.
The design so far has eighteen shards of link rows with a copy of each, a cache
in front of them, and a click path running at 360,000 events a second at the
peak. Every one of those boxes was forced by a number. None of them says what a
reader sees at the moment a machine stops answering, and that is a separate
decision which has to be made out loud, because the alternative is not having no
promise. The alternative is promising whatever the code happens to do on a path
nobody has run.
The promise is a sentence about one reader
Write it as a sentence with a reader in it and a moment in it. Somebody
shortened a link one hundred milliseconds ago and is now opening it themselves.
Somebody else is opening a link that was created a week ago while one copy of
its shard is two seconds behind. Those are different readers and they can be
given different answers at very different prices.
Here are four promises the existing eighteen shards can honour without one extra
box, and what each costs. Every number in the table is derived further down.
| the promise | what a reader sees | what it costs |
|---|---|---|
| wait for the second copy before acknowledging | never an older value, and no new links at all while a copy is unreachable | one round trip per write, and 544 writes open at once at the peak if that trip is 150 ms |
| acknowledge on the leader, read either copy | a link created a moment ago can be missing for as long as the copy is behind | nothing in latency, and 7,200 missing links during a two second stall |
| acknowledge on the leader, read both copies | never an older value | every redirect reads twice, so 72 read machines instead of 36 |
| acknowledge on the leader, read either copy, send the creator's own next read to the leader | the creator never misses their own link, everybody else can be behind | 1,200 reads a second at the leaders, 67 per shard |
Waiting for the second copy, priced
Three figures have to be said aloud because the arithmetic rests on them. A
commit to the leader's own storage takes about 1 ms. A round trip to another
machine in the same building is about 0.5 ms. A round trip to another region on
the other side of the world is about 150 ms, and no amount of money changes that
one, because it is distance.
The lesson stops here
7 more paragraphs to go
You have read the opening. The rest of the argument, the problems that check whether it landed, and the lines worth keeping at the end all come with a plan.
The first lesson of every course in the library reads the whole way through, free, so you can see exactly what the rest of them are.
See the planThe contentsThis is the reading half
Starting the course gives you your own copy of it. Every idea on every page has problems standing under it, marked with a reason rather than a tick, and any sentence you do not believe can be opened and argued with. None of that can happen on a page nobody owns.
The contents