Move data between systems so that you can prove afterwards that nothing was lost or duplicated, well enough to design the reconciliation before the pipeline, and to restart a failed run without making it worse

Moving Data Without Losing Any

Copying data between systems is where quiet corruption happens. This course covers delivery guarantees, making a step safe to repeat, catching up without stopping the source, and the count you run afterwards to prove it worked.

8 lessons, written and corrected before you arrived. Reading them here needs no account. The first reads the whole way through; the others open and then stop, because a page nobody owns cannot tell who is reading it. Starting the course gives you your own copy, where every idea has problems standing under it and you can ask about any sentence.

Start reading

01Nothing Crashed, Nothing Alerted, and the Total Is Short by Nine HundredData moving between systems fails in exactly four ways, all of them quiet. Knowing which ones a given transport can produce is what tells you which checks are worth building.
02The Sender Cannot Tell Which Half of the Journey Failedopening onlyA sender that gets no answer cannot know whether the request or the response was lost. Everything about delivery guarantees follows from that one gap, including why exactly once is not on offer.
03Run It Twice on Purpose and See What Breaksopening onlyIf the transport can deliver twice, the destination has to not care. Three techniques do this, they cost almost nothing, and one of them is wrong for your case.
04Two Writes, and the Order Between Them Is the Whole Designopening onlyA restart that resumes needs a recorded position. Save it before the work and you lose records; save it after and you repeat them. There is one way out.
05The Row Was Saved at 10:04 and Became Visible at 10:06opening onlyPolling a modified-at column is the obvious way to read changes, and it misses deletes, intermediate states, and anything written inside a slow transaction.
06Start the Stream Before You Read the Old Rowsopening onlyLoading history and following live changes are two jobs, and the join between them is where records vanish. The fix is an order, and a deliberate overlap.
07Tuesday's Number Arrived on Friday and Tuesday Is Already Publishedopening onlyA record belongs to the moment it happened and arrives whenever it arrives. The gap between those decides what you can close, and what you must restate.
08The Check Takes Four Minutes a Day and Almost Nobody Runs Itopening onlyEvery failure in this course is detectable by comparing the two ends. Counts catch two of the four, values catch the rest, and a sample is enough.