Batch and streaming
Streams process each event moments after it happens
Batch processing handles data in scheduled chunks; stream processing handles each event within seconds. Here a batch result is 14 hours old on average.
Here is one day of events. A batch job lets them pile up and processes them all at two in the morning. Simple, cheap, and each result is on average 14 hours old. Fine for a morning report. A stream processes each event moments after it arrives. That is what fraud checks, stock levels and live dashboards need. The cost is a system that never stops running, and some new problems, which this lesson is about.
Event streams
Partitions split a topic; each consumer group reads all of it
A topic is split into partitions; a consumer group shares the partitions among its members, and every group reads everything.
Streams are built on a durable log. Producers write messages with a key, here the customer. The topic is split into partitions; the key's hash picks one, so each customer's messages land in the same partition, in order, numbered by offset. The billing service runs as a group of two consumers. Each partition goes to exactly one of them, so more partitions allow more consumers working in parallel. Analytics is a separate group with one consumer. It reads every message too, at its own pace.
Event streams · Going deeper
Kafka, Kinesis and Event Hubs in numbers
Kinesis scales by shards, Event Hubs by partitions and throughput units, Kafka by partitions and brokers.
Kinesis scales by shards, Event Hubs by partitions and throughput units, Kafka by partitions and brokers. The same idea, three products. Apache Kafka (Amazon MSK, Confluent, self-run) topics of partitions replicated across brokers. Retention by time (default 7 days) or size, or compaction, which keeps the latest message per key. Consumers commit offsets per group. Amazon Kinesis Data Streams streams of shards. Each shard takes 1 MB/s or 1,000 records/s in and serves 2 MB/s out; on-demand mode adds shards automatically. Retention 24 hours by default, up to 365 days. Amazon Data Firehose delivers a stream into S3, Redshift or OpenSearch without code. Azure Event Hubs event hubs of partitions, read by consumer groups. Capacity in throughput units (1 MB/s in, 2 MB/s out each) or processing units. Retention up to 7 days on Standard, 90 on Premium and Dedicated. Speaks the Kafka protocol, and Capture writes events to storage. Ordering is guaranteed only within a partition or shard. If a total order across all events matters, there can be only one partition, and one consumer.
The same idea, three products
- Apache Kafka (Amazon MSK, Confluent, self-run) topics of partitions replicated across brokers. Retention by time (default 7 days) or size, or compaction, which keeps the latest message per key. Consumers commit offsets per group.
- Amazon Kinesis Data Streams streams of shards. Each shard takes 1 MB/s or 1,000 records/s in and serves 2 MB/s out; on-demand mode adds shards automatically. Retention 24 hours by default, up to 365 days. Amazon Data Firehose delivers a stream into S3, Redshift or OpenSearch without code.
- Azure Event Hubs event hubs of partitions, read by consumer groups. Capacity in throughput units (1 MB/s in, 2 MB/s out each) or processing units. Retention up to 7 days on Standard, 90 on Premium and Dedicated. Speaks the Kafka protocol, and Capture writes events to storage.
Offsets and replay
Each consumer group remembers its position, so the log can be replayed
Each group commits an offset; the distance to the newest message is the consumer lag; retention decides how far back you can replay.
A partition is an append-only log. New messages arrive on the right; the oldest drop off when retention expires. Billing has read up to offset 12; its lag is just 2. Analytics is at offset 5, further behind. Reading removes nothing, so both are fine. Lag is the number to watch: if it keeps growing, consumers cannot keep up. And because messages are kept, a consumer can move its offset back and replay history, after a bug fix, or to fill a new table.
Windows
The window type decides which events are counted together
Tumbling windows never overlap; hopping windows overlap; session windows end after a gap with no events.
To count a stream, you count over windows of time. Here are 10 events in fifteen minutes. Tumbling windows: fixed five-minute blocks that never overlap. 4, 4, 2. Hopping windows: ten minutes long, starting every five, so they overlap and an event can count twice: 8 and 6. Session windows: a window closes when nothing arrives for 3 minutes, the shape of one user's visit: 7, 1, 2.
Late data and watermarks
A watermark sets how long to wait for late events
Windows use event time; a watermark trailing the latest event by 10 minutes decides when a window is final and late data is dropped.
Events do not always arrive in order. A phone goes into a tunnel and sends later. So windows are built on when an event happened, and the engine keeps each window open for a while. The watermark trails the latest event time seen, here by 10 minutes. Event four happened at 12:02 but arrived at 12:09. The watermark had not passed its window, so it counts. Event six happened at 12:01 but arrived at 12:19. By then the watermark was past 12:05, the window had been finalised, and the event is dropped. A longer wait catches more, and costs memory and delay.
Exactly-once processing
Exactly-once needs a replayable source and a sink that tolerates repeats
At-least-once delivery may repeat a message after a crash; exactly-once makes the repeat harmless with an atomic or idempotent write.
Crashes happen between steps, and the order of steps decides what a crash costs. Commit the offset first and then process: a crash in between loses the message. At most once. Process first and then commit: a crash in between means the message is read and processed again. At least once, the usual default. Exactly once: write the result and the offset together, atomically, or make the write idempotent, so the second pass changes nothing. Spark does this with checkpoints and Delta tables.
Kafka durability settings · Going deeper
Three copies, two in sync, acks=all: the settings that lose nothing
With replication.factor 3, min.insync.replicas 2 and acks=all, an acknowledged message survives a broker failure.
Change data capture
CDC turns every committed change in a database into an event
Change data capture reads the database transaction log and emits each insert, update and delete as an event with before and after values.
How do you stream a database? Not by querying it over and over. Every database already writes each committed change to its transaction log, the one from lesson four. A CDC connector, such as Debezium or AWS DMS, reads that log. It publishes each change as an event: the operation, the row before and the row after. A delete arrives as an event too, which polling with a watermark would never see. Downstream, a MERGE applies the events, keeping the lakehouse copy seconds behind the database.
Applying changes · Going deeper
Apply only the latest change per key, in the source’s order
Deduplicate by key and order by a source sequence before MERGE; APPLY CHANGES (AUTO CDC) does it from a declaration.
Streaming services
A streaming platform carries, processes, stores and acts on events
Event Hubs, Kinesis and Kafka carry events; Fabric Real-Time Intelligence adds Eventstream, Eventhouse and Activator.
Each cloud assembles the same parts. A log that carries events: Event Hubs or Fabric Eventstream on Azure, Kinesis or managed Kafka on AWS, Pub/Sub on Google. An engine that processes them: Stream Analytics, Spark, Flink. A store built for fresh data over time, such as a Fabric Eventhouse, queried with KQL. And something that acts: Fabric Activator sends an alert or starts a flow when a condition occurs.
Streaming services · Going deeper
Spark Structured Streaming in practice
A Structured Streaming query reads a source, keeps progress in a checkpoint, and runs on a trigger.
A Structured Streaming query reads a source, keeps progress in a checkpoint, and runs on a trigger. One query, four decisions. Source Kafka, Event Hubs, Kinesis, a Delta table, or files through Auto Loader. Sources must be replayable so a restart can re-read from the checkpoint. Checkpoint a folder holding the offsets processed and any state (window counts, deduplication). One per query; never share or delete it, or the query starts over. Trigger the default runs micro-batches back to back; processingTime = "1 minute" runs one a minute; availableNow processes everything waiting, then stops, which gives incremental batch jobs on a schedule. Output mode and sink append writes only new rows (with a watermark for aggregates); update writes changed rows; complete rewrites the whole result. Delta tables are the usual sink, written exactly once. Stream–static and stream–stream joins. Joining a stream to a static table (orders to the customer dimension) reads the latest table version each micro-batch. Joining two streams needs watermarks on both, so the engine knows how long to remember unmatched rows.
One query, four decisions
- Source Kafka, Event Hubs, Kinesis, a Delta table, or files through Auto Loader. Sources must be replayable so a restart can re-read from the checkpoint.
- Checkpoint a folder holding the offsets processed and any state (window counts, deduplication). One per query; never share or delete it, or the query starts over.
- Trigger the default runs micro-batches back to back; processingTime = "1 minute" runs one a minute; availableNow processes everything waiting, then stops, which gives incremental batch jobs on a schedule.
- Output mode and sink append writes only new rows (with a watermark for aggregates); update writes changed rows; complete rewrites the whole result. Delta tables are the usual sink, written exactly once.
Stream–static and stream–stream joins
Lesson complete
You finished “Streaming”
Nicely done.