Counting Things as They Arrive
Nothing Can Tell You That Nothing Else Is Coming
Last timeCutting Time Into Pieces
A window has to be declared finished before it can be reported, and no evidence of completeness exists. So the system guesses, with one number, and that number is the whole trade.
Why certainty is not available
The hourly window for nine to ten cannot be reported at ten o'clock, because
events belonging to it are still arriving. So report it later. How much later?
The question has no correct answer, and the reason is worth stating precisely
rather than waving at. To know that the nine o'clock window is complete, you
would need to know that no event with an event time in that hour will ever
arrive again. There is no observation that establishes this.
Consider what you would look for. A message saying that was the last one: no
source sends one, and if it did, it could be lost. Silence from the source: a
source with nothing to say and a source whose network is down produce
identical silence. A gap in the sequence numbers: tells you something is
missing, not that nothing more is coming. The passage of time: a phone that was
in a tunnel will sync in four minutes or in four days, and nothing about the
first three days distinguishes them.
So the system decides rather than discovers, and the honest framing is that
every reported window carries an unstated assumption about what did not arrive.
The rule itself
The mechanism is one subtraction, and it is simple enough that its simplicity
is suspicious until you have seen why nothing better is available.
Keep the largest event time you have ever seen. Subtract an allowance that you
choose. Call the result the point up to which you believe you are complete; the
usual name is the watermark. Any window whose end is before that point is
declared finished, computed and emitted.
The lesson stops here
5 more paragraphs to go
You have read the opening. The rest of the argument, the problems that check whether it landed, and the lines worth keeping at the end all come with a plan.
The first lesson of every course in the library reads the whole way through, free, so you can see exactly what the rest of them are.
See the planThe contentsThis is the reading half
Starting the course gives you your own copy of it. Every idea on every page has problems standing under it, marked with a reason rather than a tick, and any sentence you do not believe can be opened and argued with. None of that can happen on a page nobody owns.
The contents