Compute over data that never stops arriving, well enough to choose a window from the question being asked, to say what your answer does when an event is late, and to keep a running total in bounded memory

Counting Things as They Arrive

Some data has no end, so there is no moment to compute over. This course covers windows, the two different clocks, how a system decides a window is finished, and what to do about the event that arrives afterwards.

8 lessons, written and corrected before you arrived. Reading them here needs no account. The first reads the whole way through; the others open and then stop, because a page nobody owns cannot tell who is reading it. Starting the course gives you your own copy, where every idea has problems standing under it and you can ask about any sentence.

Start reading

01Sort This. There Is No Last Row.Nearly everything you know about working with data assumes a final row. Remove it and half the familiar operations become impossible rather than slow.
02The Spike at Nine Was Six Hours of Traffic Arriving at Onceopening onlyAn event has a time it happened and a time it reached you. Each clock gives a wrong answer to the questions the other one owns, and both wrongs look plausible.
03Three Shapes, and the Question Tells You Which Oneopening onlyA window is a decision about which events belong together. There are three shapes in common use, the question being asked picks one, and each one costs a different amount of memory.
04Nothing Can Tell You That Nothing Else Is Comingopening onlyA window has to be declared finished before it can be reported, and no evidence of completeness exists. So the system guesses, with one number, and that number is the whole trade.
05Somebody Has Already Seen the Number You Are About to Changeopening onlyAn event arrives after its window closed. There are three things you can do with it, and the difference between them is not technical: it is what happens to a figure already published.
06An Average Needs Two Numbers, a Median Needs All of Themopening onlySome aggregates can be kept as a tiny fixed state and updated one event at a time. Others need every value you have ever seen. The difference is structural and it decides your design.
07Twelve Kilobytes Will Count a Billion Different Thingsopening onlyAn exact distinct count needs every value you have seen. An estimate within one per cent needs a fixed twelve kilobytes, and the reasoning that gets you there is worth following once.
08The Windows Came Back, and So Did Nine Minutes of Totalsopening onlyEverything in this course lives in memory on a machine that will be restarted. Surviving that means saving state and position together, and it means the output system has to cooperate.