Answers · The systems around the model
Why can't a system just stay consistent and available when the network splits?
In shortDuring a split, a cut-off machine can only answer from the data it already holds, so it must either risk being wrong or refuse to answer.
Because a machine that has been cut off from the others has exactly one source of information left: the data it already holds. Answer from that and the reply may be out of date, or a write may be accepted that later collides with one taken on the other side; refuse to answer and a perfectly healthy machine becomes unavailable to every client that can still reach it. There is no third thing to consult, so there is no third behaviour.
Put numbers on it. A service takes 35 requests a second, and the network splits for ten minutes.
Those 21,000 arrive on the smaller side alone, from clients who know nothing about the split and cannot be told to come back later. Under a refusing policy all 21,000 fail and the recovery afterwards is clean: there is nothing to merge. Under an answering policy none fail, but every reply was produced without news of the other side. Suppose one request in six is a write, so 35/6 is about 5.8 writes a second on that side, and over 600 seconds that is roughly 3,500 writes that were accepted without the other side's knowledge and have to be reconciled once the link returns. Twenty one thousand refusals against three and a half thousand conflicts. Neither column is empty, and which one you would rather have is a question about what the service is for, not about how well it is built.
The guess that fails
The natural guess is that this is an engineering shortfall: add replicas, use a faster agreement protocol, merge writes cleverly, and a good system gets both properties. It cannot, because agreement itself needs a majority, and a majority is precisely what a split takes away from at least one side. No amount of cleverness creates a source of truth that the network is currently preventing a machine from reaching; the machine is being asked to answer a question whose answer it cannot obtain. The choice is forced by physics, not by design taste, and it is forced before the split happens, because during the split there is nobody to ask.
The arithmetic of majorities shows how sharply this bites. With five machines, a majority is 3, since 5/2 is 2.5 and you need strictly more than half. A 3/2 split therefore leaves the group of three able to decide and the group of two unable to, so one side keeps serving and the other refuses. With four machines a majority is also 3, because 4/2 is 2 and strictly more than 2 means 3: a 2/2 split leaves neither side able to decide, and both stop. Changing the cluster size from five to four turned a half-outage into a total one, without changing a line of code.
The choice belongs to the operation
What a system does about this is not one setting. Showing a product page slightly out of date costs almost nothing and refusing costs a sale, so serve it. Two orders accepted for the last item in stock cost a refund and an apology, which is usually cheaper than ten minutes of refusing orders, so accept and reconcile. A password change accepted on both sides of a split leaves a credential working that its owner believes they revoked, which is a cost no availability pays for, so refuse it unless the machine is in the larger group.
| operation | answer from a possibly stale copy | accept the write during the split | what being wrong costs |
|---|---|---|---|
| product page | yes | not applicable | a slightly old price, corrected within seconds |
| place an order | yes | yes, reconciled afterwards | occasional duplicate order, refunded; roughly 1 in 300 of the 3,500 |
| change a password | no | no, unless in the larger group | a revoked credential still working |
The version of this trade you actually pay for is the quiet one. Keeping copies in agreement costs a round trip whether or not anything has failed, and that bill arrives every second of every day, while the partition arrives a handful of times a year.
Here is the local work of finding and returning the value and is the one way delay to the furthest copy that must be consulted, so is the round trip. With local work of 2 ms and a one way delay of 70 ms, the client waits 2 + 140 = 142 ms for an answer it can rely on. Served from a copy next door, the same read takes 2 ms. That is 71 times slower, and 142 ms is most of a 200 ms page load budget, which is why systems spread across continents serve reads from a nearby copy and accept that the answer may be a little old. The design is shaped by the everyday round trip, not by the rare outage.
Two things change the picture. If all the copies sit in one building, a split is not rare but effectively nonexistent, so the choice never gets exercised and you pay only the round trip. And if the writes for an operation can be merged without loss, say by counting rather than overwriting, the reconciliation after a split costs almost nothing, though both sides still had to answer from what they held. What does not change is that a machine which cannot reach the others can only speak for itself. Write the rule for each operation down in advance, choose it by weighing being wrong against being unavailable, and expect the common compromise, reads everywhere and writes in the larger group, to be right more often than either extreme.
Worth keeping
- A machine cut off by a network split can answer from what it holds or refuse; there is no third option because there is no third source of information.
- A majority is strictly more than half the machines, so five machines survive a 3/2 split with one side serving while four machines split 2/2 leave neither side able to decide.
- Keeping copies in agreement costs a round trip on every read whether or not anything has failed, and that everyday cost shapes most systems more than the rare partition does.
Where this goes next
This question is one step of When One Machine Is Not Enough, a course that works through its subject from the beginning. The page that teaches this part of it is The Choice a Partition Forces, and it reads the whole way through whether or not you have an account.
What an account changes is the practice: problems standing under every idea, marked with a reason rather than a tick, in a copy of the course that is yours.
Read that pageThe whole courseOther questions people ask about this
- What does it actually mean to say a copy will be 'eventually' consistent?It means the copies will agree in the end, and that for some unspecified time a read may return an out of date value.
- What's the difference between a CDN cache and caching inside my own app?A CDN cache sits near the user and can only hold answers that are identical for everyone, while a cache inside your app can be personal but exists once per process.
- What's the difference between LRU and LFU, and which one should I use?LRU drops the item unused for longest; LFU drops the item requested least often. Both guess at future use, and the cheap win is refusing one-off arrivals rather than counting requests.
- How can I tell if my traffic is bursty or just high volume?Compare the variance of the gaps between arrivals with the square of their average gap: high volume alone does not make traffic bursty.
- Why does average latency hide the slow requests users actually feel?Waiting times are lopsided, so the average sits between the common case and the bad case and describes neither, and it is the bad case that users keep meeting.
- Should the gradients be summed or averaged across GPUs?Sum the gradients when each device already divides its loss by the global batch size; average them when each device uses its own local mean. Doing both makes the gradient too small by the number of devices.