Counting Things as They Arrive
Three Shapes, and the Question Tells You Which One
Last timeTwo Different Clocks
A window is a decision about which events belong together. There are three shapes in common use, the question being asked picks one, and each one costs a different amount of memory.
The three shapes
Since no aggregate is possible over an input with no end, every streaming
aggregate is over a window, and a window is a rule saying which events belong
together. Three rules cover almost everything built in practice.
A fixed window, also called tumbling, cuts time into adjacent equal pieces. Nine
to ten, ten to eleven, eleven to twelve. The pieces do not overlap and they
leave no gaps, so each event belongs to exactly one of them. This is the shape
everybody draws first and the shape most reports are written against.
A sliding window has a width and an advance, and the width is larger than the
advance. A one-hour window advancing every five minutes gives you a window
starting at 9:00, another at 9:05, another at 9:10, each one an hour long. They
overlap heavily, which means a single event belongs to twelve of them at once.
A session window has no boundaries until the data supplies them. You specify a
gap, say thirty minutes. A window opens on the first event for a key, extends
with each new event, and closes when the gap passes with nothing arriving. Two
sessions for the same key can be four minutes long and nine hours long, and
neither has anything to do with the clock on the wall.
Reading the shape off the question
The shape is not a preference. It is determined by the question, and the words
of the question usually give it away.
Per hour, per day, per calendar month means fixed. The question is about a
named period that somebody else defined, usually for reporting or billing, and
the periods must tile without overlap so that the parts add to the whole.
In the last hour, asked repeatedly, means sliding. The question is about a
quantity that should be current rather than about a named period. Alerting is
the clearest case. Nobody wants to know the error rate for the 10:00 hour at
11:00; they want to know the error rate over the last five minutes, now, and
again thirty seconds from now. A fixed window used here produces an alert that
is on average half a window out of date and resets its count at a boundary for
no reason connected to anything real.
Per visit, per conversation, per trip, per editing burst means session. The
question is about an episode of activity whose length is a property of the
behaviour rather than of the clock.
| fixed | sliding | session | |
|---|---|---|---|
| sales in each calendar d | 1 | 0 | 0 |
| invoice totals per month | 1 | 0 | 0 |
| error rate right now | 0 | 1 | 0 |
| is traffic unusual this | 0 | 1 | 0 |
| pages read per visit | 0 | 0 | 1 |
| length of a support conv | 0 | 0 | 1 |
| distance of one delivery | 0 | 0 | 1 |
The session mistake deserves a sentence of its own, because it is silent. If
visits are measured in fixed hourly windows, a visit from 10:50 to 11:10 is
recorded as two visits of ten minutes each. Nothing errors, the number is
plausible, and the reported average visit length is wrong in a direction that
depends on your window size.
The lesson stops here
4 more paragraphs to go
You have read the opening. The rest of the argument, the problems that check whether it landed, and the lines worth keeping at the end all come with a plan.
The first lesson of every course in the library reads the whole way through, free, so you can see exactly what the rest of them are.
See the planThe contentsThis is the reading half
Starting the course gives you your own copy of it. Every idea on every page has problems standing under it, marked with a reason rather than a tick, and any sentence you do not believe can be opened and argued with. None of that can happen on a page nobody owns.
The contents