Answers · Where the data lives
How do I make a step safe to run again when I'm not sure whether it already ran?
In shortChange the operation so that sending it twice leaves the same stored data as sending it once, instead of relying on remembering not to send it again.
Change the operation so that a second delivery leaves the same stored state as the first. The fix is in the destination, not in your memory of whether the step already ran: a retry will happen, and the destination has to be indifferent to it.
- the error in the stored total
- the number of deliveries
- the value the message carried
Take a balance of 100 and a message that adds 40. After one delivery the balance is 140. After a second identical delivery it is 180, so the error is , which is . If a timeout causes eleven deliveries the error is . The error grows with every extra delivery and does not correct itself. The same message carrying a setting of 140 leaves 140 after one delivery and 140 after two, because the operation replaces the value instead of adding to it. That is the difference between an add and a set, and it is why rewriting a message to carry the full new value is often the cheapest fix.
Reject the repeat at the destination
For an add, the destination needs something in its stored state to recognise the repeat. A remembered key does this: the sender attaches a key identifying the operation, not the attempt, and every retry carries the same key. The insert of that key and the work must sit in one transaction. If the key insert succeeds, do the work; if it fails because the key is already there, skip the work. A key inserted outside the transaction is the common failure: a crash after the key and before the work loses the work, and a crash after the work and before the key permits the duplicate. The key must be kept at least as long as the longest possible retry, then cleaned up.
A conditional write needs no extra table. With account 51 at version 7, run update balances set amount = 140, version = 8 where account = 51 and version = 7. The first delivery changes one row and leaves version 8. The second matches no row at version 7 and changes nothing. Zero rows changed on a retry is the signal that you already did it. A state machine uses the same idea with named stages: update orders set status = shipped where id = 904 and status = packed. After the first delivery the row is shipped, so the second matches nothing. It also rejects a message that arrives out of order, which the other two do not.
| step | technique | state before | second delivery does | state after | what happened |
|---|---|---|---|---|---|
| 1 | none | balance 100 | adds 40 again | balance 180, wrong | Nothing rejected the second delivery because nothing was looking. |
| 2 | remembered key | balance 140, key stored | insert of the key fails, work skipped | balance 140, correct | The key and the work committed together. |
| 3 | conditional write | balance 140, version 8 | update matches no row at version 7 | balance 140, correct | The version column already held the memory. |
| 4 | state machine | order shipped | transition from packed matches nothing | order shipped, correct | The states were the memory, and an out of order message is rejected too. |
The natural guess is to remember not to re-run the step. That fails because the memory is usually in the wrong place. A local sent flag written before the send loses the message if the process crashes between the flag and the send; written after the send, it permits a duplicate if the crash falls in between. A key generated per attempt rather than per operation passes a code read and fails under retry. And a check followed by a write is not atomic unless the destination enforces it, so two concurrent deliveries can both pass the check. The destination must reject the second effect, not merely be accompanied by a note that it should not happen.
Where this stops
Anything whose effect leaves your system cannot be made safe by code you control alone: an email, a card charge, a message that opens a door. The thing that would have to remember is somebody else's. Ask whether the far end accepts a key of your choosing and refuses to act twice on it. Payment providers often do, and then the hard case collapses into the easy one. Where the far end does not, record locally that you sent it before sending, and accept that a crash between record and send loses the message rather than duplicating it. That is at most once, chosen deliberately for the step where a duplicate is worse than a loss.
Safe to repeat is exact, not approximate. A step that survives two deliveries but not eleven has not been made safe, only made lucky. The test is to build the message, deliver it, deliver the identical message again, assert that the stored state is exactly what it was after the first delivery, then deliver it a third time. Assert the balance is 140, not that a key table has one row. Then deliver two copies concurrently, because a remembered key without a transaction or a conditional write against a stale read passes the sequential test and fails that one. A step that is safe to repeat makes a restart harmless; it does not make the restart cheap, which needs a separate record of where the run got to.
Worth keeping
- Running twice leaves the same state as running once only when the destination rejects the repeat, and that is a property of the stored data, not of the sender's intention.
- A remembered key belongs in the same transaction as the work it guards; a key outside it trades a duplicate for a lost message.
- For an effect outside your system, use the far end's idempotency key if it has one; otherwise choose loss over duplication deliberately.
Where this goes next
This question is one step of Moving Data Without Losing Any, a course that works through its subject from the beginning. The page that teaches this part of it is Making a Step Safe to Repeat, and it reads the whole way through whether or not you have an account.
What an account changes is the practice: problems standing under every idea, marked with a reason rather than a tick, in a copy of the course that is yours.
Read that pageThe whole courseOther questions people ask about this
- How can I prove nothing was lost without comparing every single row?Sample a few thousand rows drawn from the source, look each one up in the destination, and add one total row count. That proves loss and duplication without reading every row.
- Why is the date off by one day after I save it and read it back?You saved a calendar date as if it were a precise moment, so showing it in another time zone can move it to the day before or after.
- Is UTF-8 the same thing as Unicode, or what's the difference?No: Unicode assigns a number to each character, while UTF-8 is a way of writing those numbers as bytes, using one to four bytes per character.
- Why does my CSV file fall apart when one of the fields has a comma in it?Because the same comma both separates fields and sits inside the value, so a reader splitting on commas finds one field too many and every later column shifts.
- Why do programmers write numbers in hexadecimal instead of just normal numbers?Programmers use hexadecimal because each digit stands for exactly four bits, so one byte is always written as two digits and the columns line up.
- Why does my count come out doubled after the job restarts?Your count doubled because the restart replayed events the saved counter already held: the state and input position were not saved as one snapshot.