Late data in Flink: allowed lateness or a side output?
Event-time windows in Flink close when the watermark passes the window end. Anything that arrives after that is late, and by default it is dropped without a word. There are two ways to keep it: let the window accept stragglers for a while, or route them somewhere else.
Start with the watermark
Most lateness problems are really watermark problems. A bounded out-of-orderness strategy says "assume events are at most N seconds out of order":
WatermarkStrategy<Event> wm = WatermarkStrategy
.<Event>forBoundedOutOfOrderness(Duration.ofSeconds(30))
.withTimestampAssigner((e, ts) -> e.getEventTime())
.withIdleness(Duration.ofMinutes(1));
withIdleness matters more than people expect. A Kafka partition that stops receiving data holds back the watermark for the whole job; marking it idle after a minute lets the others move on.
Allowed lateness
events
.keyBy(Event::getAccountId)
.window(TumblingEventTimeWindows.of(Duration.ofMinutes(5)))
.allowedLateness(Duration.ofMinutes(10))
.aggregate(new CountAgg());
The window fires on time, then fires again for every late event that arrives within ten minutes. Downstream sees updated results for the same key and window, so the sink has to upsert. In exchange, results converge without extra code.
The cost is state: every window stays in the state backend for ten extra minutes. With many keys and short windows, that adds up quickly.
Side output
OutputTag<Event> late = new OutputTag<Event>("late") {};
SingleOutputStreamOperator<Count> counts = events
.keyBy(Event::getAccountId)
.window(TumblingEventTimeWindows.of(Duration.ofMinutes(5)))
.sideOutputLateData(late)
.aggregate(new CountAgg());
counts.getSideOutput(late).sinkTo(lateSink);
Windows close on time and leave state immediately. Late events go to a separate stream, often a table that a batch job reconciles overnight.
How I choose
If the output feeds a dashboard or an alert, I use allowed lateness: people expect the number to become correct on its own. If it feeds an append-only sink, or late volume is tiny, I use a side output and reconcile in batch. Doing both is legitimate too: allow a short lateness and send anything later to the side output.