ContentsThe library

When One Machine Is Not Enough

Which Thing Happened First

Last timeGetting Machines To Agree

Separate machines cannot agree on what time it is, so ordering events by their timestamps quietly produces wrong answers, and the fix is to count rather than to read a clock.

Two writes arrive for the same record. One carries the timestamp 14:02:07.113 and

the other 14:02:07.109. The second one is kept because it is later. This is wrong

often enough to matter, and the reason is that the two numbers were read from two

different clocks.

Clocks disagree, and they lie about it

A machine's clock drifts because its oscillator is imperfect. Corrections arrive

over the network, which has an unknown delay, so the correction itself is

approximate. The result is that two machines in the same rack commonly read tens

of milliseconds apart.

FIG 1How far a clock can wander between corrections
0.0037.5075.00112.50150.001.0150.8300.5450.3600.0seconds since the last correction
an ordinary machine clocka well maintained one
Ten minutes between corrections is enough for an ordinary clock to drift further than the gap between most pairs of events you might want to order. The lower line is what careful operation achieves, and it is still not zero.

Worse, a clock can move backwards. When a correction decides the machine is

ahead, the clock is set back, and an event happening after another can be stamped

with an earlier time. Any logic that assumes timestamps increase has a bug

waiting for that moment.

The lesson stops here

7 more paragraphs to go

You have read the opening. The rest of the argument, the problems that check whether it landed, and the lines worth keeping at the end all come with a plan.

The first lesson of every course in the library reads the whole way through, free, so you can see exactly what the rest of them are.

See the planThe contents

This is the reading half

Starting the course gives you your own copy of it. Every idea on every page has problems standing under it, marked with a reason rather than a tick, and any sentence you do not believe can be opened and argued with. None of that can happen on a page nobody owns.

The contents

The rest of this course

  1. 01The Three Reasons, and What Each One Costs
  2. 02How Far Behind a Copy Is Allowed to Beopening only
  3. 03Silence Means Nothing In Particularopening only
  4. 04What You Give Up When the Network Splitsopening only
  5. 05How a Group Decides Without a Bossopening only
  6. 06Which Thing Happened Firstyou are here
  7. 07Cutting the Data Into Piecesopening only
  8. 08Designing for the Failure You Will Actually Getopening only

Read alongside