Moving Data Without Losing Any
Run It Twice on Purpose and See What Breaks
Last timeAt Least Once, At Most Once
If the transport can deliver twice, the destination has to not care. Three techniques do this, they cost almost nothing, and one of them is wrong for your case.
What safe to repeat means
The last lesson ended somewhere specific. The transport will deliver at least
once, which means it will sometimes deliver twice, and no amount of work on the
transport changes that. So the work moves to the destination, and the property
the destination needs is this: running the step twice leaves the system in the
same state as running it once.
Three words in that sentence are doing real work.
State, not return value. The second delivery may well return something
different, perhaps a note saying it was already applied, and that is fine. What
must not differ is what anybody looking at the data afterwards can see.
Same, not similar. A total that is right to the penny after one delivery and
wrong by one unit after two is not safe to repeat. The property is exact or it
is absent.
And twice includes every number above twice. A retry loop with a bad timeout can
deliver the same message eleven times, and a step that survives two deliveries
and not eleven has not been made safe, only made lucky.
- the error in the stored total
- how many times the message was delivered
- the value the message carried
The easy ones and the hard ones
Operations sort into three groups, and the sorting rule is whether the result
depends on what was there before.
Setting a value is already safe. Store the temperature as 21 degrees and store
it again and it is 21 degrees. Nothing needs doing, and a surprising share of a
pipeline turns out to be of this shape once the messages carry the full new
value rather than a change to the old one.
Adding is not safe. Add 40 to the balance twice and the balance is wrong by 40,
and the amount of wrongness is exactly the thing that will be noticed and
exactly the thing nobody can reconstruct six weeks later.
And then the hard group: anything whose effect leaves your system. An email to
a customer. A charge on a card. A message to a device that opens a door. These
cannot be made safe by anything you write, because the thing that would have to
remember is somebody else's.
| already safe | needs a remembered key | needs a state rule | depends on someone else | |
|---|---|---|---|---|
| store a reading | 1 | 0 | 0 | 0 |
| add to a running total | 0 | 1 | 0 | 0 |
| replace a profile | 1 | 0 | 0 | 0 |
| increment a counter | 0 | 1 | 0 | 0 |
| advance an order to ship | 0 | 0 | 1 | 0 |
| email a receipt | 0 | 0 | 0 | 1 |
| charge a card | 0 | 0 | 0 | 1 |
For the last group there is still something to do, and it is worth saying
before the techniques, because it changes what you build. Ask whether the far
end will accept a key of your choosing and refuse to act twice on it. Payment
providers do. Several messaging services do. Where the answer is yes, the hard
case collapses into the easy one and the far end does the remembering. Where the
answer is no, the honest design is to record locally that you sent it, before
sending, and accept that a crash between the record and the send loses the
message rather than duplicating it: at most once, chosen deliberately, for the
one step where a duplicate is worse than a loss.
The lesson stops here
2 more paragraphs to go
You have read the opening. The rest of the argument, the problems that check whether it landed, and the lines worth keeping at the end all come with a plan.
The first lesson of every course in the library reads the whole way through, free, so you can see exactly what the rest of them are.
See the planThe contentsThis is the reading half
Starting the course gives you your own copy of it. Every idea on every page has problems standing under it, marked with a reason rather than a tick, and any sentence you do not believe can be opened and argued with. None of that can happen on a page nobody owns.
The contents