Moving Data Without Losing Any
Tuesday's Number Arrived on Friday and Tuesday Is Already Published
Last timeThe Old Data and the New at Once
A record belongs to the moment it happened and arrives whenever it arrives. The gap between those decides what you can close, and what you must restate.
When it happened, when it arrived
Every record has two times attached to it and a great many reporting problems
come from a system that only wrote down one.
There is the time the thing happened: the sale was made, the sensor read, the
button pressed. And there is the time it reached you: the message was consumed,
the row was written, the file landed.
For most records the two are seconds apart and nobody notices the difference.
For some they are days apart, because a phone was in a tunnel, a batch job
failed and was rerun, a partner sends on Fridays, or one of the six hours of
replay from the fourth lesson happened to fall across midnight.
Grouping by arrival time is easy, requires nothing to be recorded, and is wrong
in a way that is hard to undo. Tuesday is short by the sales that arrived late
and Wednesday is long by the same amount. Both days are wrong, both are
plausible, and the error moves with the next outage rather than staying still.
Grouping by event time is right and it costs something: you cannot finish
Tuesday at midnight, because records belonging to Tuesday are still coming.
Three rules make this tractable.
Carry the event time on the record, set by whatever observed the event, and never
reconstruct it later from the arrival. Keep the arrival time too, as a separate
field, because the difference between them is the only data you will ever have
about lateness. And when anybody asks for a number by day, establish which of the
two clocks they mean before answering, because they almost certainly have not
considered that there are two.
Measuring lateness
The question is how long to wait before declaring a window closed, and it is
usually answered by somebody picking a number that sounds reasonable. It does not
have to be a guess. The difference between arrival and event time is a
measurement you already have, for every record you have ever received.
The lesson stops here
3 more paragraphs to go
You have read the opening. The rest of the argument, the problems that check whether it landed, and the lines worth keeping at the end all come with a plan.
The first lesson of every course in the library reads the whole way through, free, so you can see exactly what the rest of them are.
See the planThe contentsThis is the reading half
Starting the course gives you your own copy of it. Every idea on every page has problems standing under it, marked with a reason rather than a tick, and any sentence you do not believe can be opened and argued with. None of that can happen on a page nobody owns.
The contents