Answers · Where the data lives
Why does my count come out doubled after the job restarts?
In shortYour count doubled because the restart replayed events the saved counter already held: the state and input position were not saved as one snapshot.
Your count is doubled because the restart replayed events that the saved counter already contained. The checkpointed state and the saved input position did not describe the same instant, so the job resumed from an earlier position and folded the same window in a second time. Saving state on its own does not prevent this. What has to survive together is the window state, every input position and the pending timers, because any two without the third leave a resumable-looking job that is not resumable.
Take a window that opens at 10:06. By 10:08 it has seen ten events, so its accumulator is ten. The state is written at 10:08, which is after those ten events have been folded in. The input position is written at 10:06, before any of them. A crash between the two writes leaves the state ahead of the position. On restart the source seeks to 10:06, the ten events arrive again, and each one increments the accumulator a second time. The counter finishes at twenty. The same mistake with the writes in the other order gives the opposite symptom: the position is ahead of the state, nothing is replayed, and the total is short by ten. Neither case produces an error. The number is simply wrong in a window somebody has probably already read.
The natural guess is that saving the state is the hard part and that the position is bookkeeping. It is the other way round. A state without a position cannot be continued, because nothing says where to start reading, and a position without the matching state either skips work or repeats it. The two must be one snapshot, so that restoring the snapshot puts the job in the condition where resuming from those positions reproduces exactly those accumulators. Saving them as two writes gives a crash a choice between losing windows and double counting them.
| step | scheme | what was saved | what resumes | counter after restart | what happened |
|---|---|---|---|---|---|
| 1 | state first, then position | state at 10:08 with counter 10, position at 10:06 | replays 10:06 to 10:08, ten events | 10 + 10 = 20 | The accumulator already contained the ten events, so the replay adds them again. |
| 2 | position first, then state | position at 10:08, state at 10:06 with counter 0 | replays nothing | 0 | The ten events were acknowledged as read and the counter holding them was lost, so the total is short by ten. |
| 3 | one snapshot at 10:06 | state and position both at 10:06 with counter 0 | replays 10:06 to 10:08 once | 0 + 10 = 10 | Correct internal total, at the cost of redoing two minutes of work. |
| 4 | one snapshot at 10:06, output already sent | state and position at 10:06; the window wrote 5 at 10:07 | replays 10:06 to 10:07 | internal 5, destination two rows of 5 | The checkpoint cannot recall the write, so the destination needs a keyed replace or a transaction tied to the checkpoint. |
A single snapshot does not mean pausing the stream. A marker is injected at the sources and travels with the events. When an operator has seen the marker on all its inputs, it writes its own state and passes the marker on. The set of states written for one marker is a consistent cut: everything before the marker is included everywhere, everything after it is included nowhere. Restoring that cut resumes from positions that match the accumulators exactly, so the replay repeats work but does not double count.
The output has already left
A consistent snapshot fixes the inside of the job. It does nothing to a result that has already been sent. If a window total of five was written to a database at 10:07, and the job restarts from a 10:06 snapshot, the window closes again at 10:07 and writes five again. The inside of the job is correct and the destination has two copies. No checkpoint can recall a row that has already been committed, an email that has already been sent or a webhook that has already been delivered. This is why the checkpoint and the output write have to agree, and why the agreement cannot be arranged inside the streaming engine alone.
The guarantee people call exactly once is a property of the job together with the destination, not of the job by itself. There are two ways to get it. The write can be repeatable without changing the result: write the total for a window into a row keyed by that window, so doing it twice leaves the same row. That requires only a keyed replace, which most destinations support. Or the write can be held open and committed only when the checkpoint covering it succeeds. A restart rolls back the uncommitted write along with the state, so nothing was visible twice. That costs latency equal to the checkpoint interval and needs a destination that can hold a transaction across a checkpoint boundary.
Where it stops being true
An outbound email, a webhook or an append-only log gives at least once at best, because a sent message cannot be unsent. The receiver has to handle repeats. The trap is a counter increment. Incrementing a counter looks like a write and is not repeatable, since doing it twice gives a different answer, so a job that reports by incrementing is unfixable at the destination and has to be changed to write a total instead. Decide what the sink is before deciding the rest, because no work inside the job creates a guarantee the destination cannot express.
Worth keeping
- A checkpoint is resumable only when window state, input positions and pending timers are saved as one object.
- Saving position before state makes totals short, not doubled, because the replay is skipped rather than repeated.
- Exactly once is a property of the destination as well as the job, so the write must be repeatable or committed with the checkpoint.
Where this goes next
This question is one step of Counting Things as They Arrive, a course that works through its subject from the beginning. The page that teaches this part of it is Surviving a Restart, and it reads the whole way through whether or not you have an account.
What an account changes is the practice: problems standing under every idea, marked with a reason rather than a tick, in a copy of the course that is yours.
Read that pageThe whole courseOther questions people ask about this
- Why can't I just average the old average with the new number to get a running average?Averaging the old average with the new number gives that single new value half the weight, when it deserves one share out of however many values came before.
- Should I use a sliding window or a fixed window?Use a fixed window when the question names a period that must add up, and a sliding one when it asks about the recent past as of now. Questions about episodes want neither.
- How do you count unique visitors without remembering every visitor?Turn each visitor identifier into a random-looking code, keep only a small summary of the rarest pattern seen, and scale it up; repeats change nothing and memory stays fixed.
- How does a streaming system know when a window is finished?It cannot know. It guesses: it takes the latest event time it has seen, subtracts an allowance you choose, and treats every window ending before that point as finished.
- Why is the date off by one day after I save it and read it back?You saved a calendar date as if it were a precise moment, so showing it in another time zone can move it to the day before or after.
- Is UTF-8 the same thing as Unicode, or what's the difference?No: Unicode assigns a number to each character, while UTF-8 is a way of writing those numbers as bytes, using one to four bytes per character.