Moving Data Without Losing Any
Two Writes, and the Order Between Them Is the Whole Design
Last timeMaking a Step Safe to Repeat
A restart that resumes needs a recorded position. Save it before the work and you lose records; save it after and you repeat them. There is one way out.
What a position actually is
The last lesson made a step safe to repeat, which makes a restart harmless. It
does not make a restart cheap. A worker that comes back and reprocesses six
hours of history is correct and useless, and the cost is not only time: it is
load on the source, on the destination, and on whatever else shares them.
So the worker needs to record where it got to. The record exists to answer
exactly one question, and keeping that question in view settles most of the
design decisions that follow: if this process died right now and came back,
where should it start?
A position therefore has to name a point in the data such that everything before
it is finished and everything after it is not. That phrasing rules out two of
the three obvious candidates.
An identifier will not do, because identifiers are not ordered. Knowing that
record 8841 is done says nothing about 8840 unless the identifiers were handed
out in processing order, which is a property almost nothing guarantees.
A timestamp will not quite do either, and this is the one that catches people,
because a timestamp looks ordered. Two records can carry the same timestamp. A
worker that records the last timestamp it saw and resumes strictly after it
loses every record that shared that moment, and a worker that resumes at it
repeats them. Second resolution makes this common, millisecond resolution makes
it rare and not absent, and at volume rare means daily.
- records reprocessed after a restart
- the record rate
- how far back the worker has to start
- the number of workers restarting
What does work is a sequence number from the source, or a timestamp paired with
a tie-breaker, or a composite of the two. The shape matters less than the
property: for any point, the data falls cleanly into done and not done, with
nothing ambiguous on the boundary.
Before or after, both wrong
Now the part that is genuinely a design problem rather than a detail. There are
two things to write: the result of the work, and the new position. They are two
separate writes, and a process can die between any two writes.
The lesson stops here
2 more paragraphs to go
You have read the opening. The rest of the argument, the problems that check whether it landed, and the lines worth keeping at the end all come with a plan.
The first lesson of every course in the library reads the whole way through, free, so you can see exactly what the rest of them are.
See the planThe contentsThis is the reading half
Starting the course gives you your own copy of it. Every idea on every page has problems standing under it, marked with a reason rather than a tick, and any sentence you do not believe can be opened and argued with. None of that can happen on a page nobody owns.
The contents