Batch and streaming

Streams process each event moments after it happens

00:0006:0012:0018:0000:00eventsBatchevents wait in storage until the nightly runprocessed together at 02:00: on average 14 hours oldStreameach processed within seconds of arrivingBatch handles data in scheduled chunks; streaming handles each event moments after it happens.In this day of 22 events, a nightly batch delivers each result 14 hours late on average; a stream delivers it in seconds. Batch is simpler, cheaperper record and fine for reports read the next morning. Streaming suits anything that must react now: fraud checks, stock levels, sensors, livedashboards. Spark Structured Streaming runs a stream as a series of small batches, and can also run it once, like a batch job.

Batch processing handles data in scheduled chunks; stream processing handles each event within seconds. Here a batch result is 14 hours old on average.

Event streams

Partitions split a topic; each consumer group reads all of it

producerkey = customertopic: ordersp0c90c41c92c43p1c10p2c70c21c22c73c24c55c76offset under each message · same key, same partition, in orderconsumer group: billingc1p0, p1c2p2consumer group: analyticsc1 reads p0, p1, p2A topic is split into partitions; each consumer group reads every message, sharing the partitions among its members.Messages with the same key always hash to the same partition, so one customer’s events stay in order (here 12 messages, partitions of 4, 1, 7).Within a group each partition goes to exactly one consumer, so partitions set the maximum parallelism. Groups are independent: billing andanalytics each read everything at their own pace. This is Kafka’s model; Event Hubs and Kinesis (shards) work the same way.

A topic is split into partitions; a consumer group shares the partitions among its members, and every group reads everything.

Event streams · Going deeper

Kafka, Kinesis and Event Hubs in numbers

Kinesis scales by shards, Event Hubs by partitions and throughput units, Kafka by partitions and brokers.

The same idea, three products

  • Apache Kafka (Amazon MSK, Confluent, self-run) topics of partitions replicated across brokers. Retention by time (default 7 days) or size, or compaction, which keeps the latest message per key. Consumers commit offsets per group.
  • Amazon Kinesis Data Streams streams of shards. Each shard takes 1 MB/s or 1,000 records/s in and serves 2 MB/s out; on-demand mode adds shards automatically. Retention 24 hours by default, up to 365 days. Amazon Data Firehose delivers a stream into S3, Redshift or OpenSearch without code.
  • Azure Event Hubs event hubs of partitions, read by consumer groups. Capacity in throughput units (1 MB/s in, 2 MB/s out each) or processing units. Retention up to 7 days on Standard, 90 on Premium and Dedicated. Speaks the Kafka protocol, and Capture writes events to storage.
Ordering is guaranteed only within a partition or shard. If a total order across all events matters, there can be only one partition, and one consumer.

Offsets and replay

Each consumer group remembers its position, so the log can be replayed

Partition p0 of orders: an append-only log012345678910111213past retention: deletednewbilling at offset 12lag 2analytics at offset 5lag 9: still catching upreset an offset to replay history still retainedA stream is a log that consumers read at their own pace, each remembering its position as an offset.Reading does not remove a message: billing and analytics read the same log, each committing the offset it has reached. The gap to the newestmessage is the consumer lag, the number to watch. Messages stay until retention expires (Kafka’s default is 7 days), so a new consumer, or afixed bug, can rewind and replay. That replayability is what makes a stream a reliable source rather than a fire hose.

Each group commits an offset; the distance to the newest message is the consumer lag; retention decides how far back you can replay.

Windows

The window type decides which events are counted together

12:0012:0512:1012:15eventsTumbling 5 min442Hopping 10 min, every 586Session, 3-min gap712counts: 4, 4, 2 · 8, 6 · 7, 1, 2Streaming aggregates are computed over windows of time, and the window type decides which events count together.Tumbling windows are fixed and never overlap: orders per 5 minutes. Hopping windows have a fixed size but start more often, so they overlap: a10-minute average updated every 5 minutes (Spark calls this a sliding window). Session windows close after a gap with no events: one user visit.Azure Stream Analytics, Spark and Flink all offer these. Counts computed from the 10 events shown; the third tumbling window ends at 12:15.

Tumbling windows never overlap; hopping windows overlap; session windows end after a gap with no events.

Late data and watermarks

A watermark sets how long to wait for late events

.withWatermark('event_time', '10 minutes').groupBy(window('event_time', '5 minutes')).count()eventhappenedarrivedwatermarkwindowresulte112:0012:00—12:00–12:05countede212:0312:03—12:00–12:05countede312:0712:07—12:05–12:10countede412:0212:09—12:00–12:05countede512:1612:16—12:15–12:20countede612:0112:1912:0612:00–12:05dropped: window closede4: 7 minutes late, but itswindow is still opene6: 18 minutes late; thewatermark passed its windowA watermark tells the engine how long to wait for late events before it finalises a window and frees its memory.Events are grouped by when they happened (event time), not when they arrived: phones go offline, networks retry. The watermark trails thelatest event time seen by a set delay, here 10 minutes. A window stays open until the watermark passes its end; later arrivals for it aredropped. A longer delay accepts later data but holds more state and publishes final results later. Decisions computed from the events shown.

Windows use event time; a watermark trailing the latest event by 10 minutes decides when a window is final and late data is dropped.

Exactly-once processing

Exactly-once needs a replayable source and a sink that tolerates repeats

At most oncecommit offset, then processreadcommitprocess✗crash at ✗: message lostAt least onceprocess, then commit offsetreadprocesscommit✗crash at ✗: processed againExactly onceprocess and commit together, or idempotent sinkreadprocesswrite + offset, atomiccrash anywhere: same resultExactly-once means each event affects the result once, even across crashes: replay plus a sink that tolerates repeats.Most systems deliver at least once: after a crash they re-read from the last committed offset, so some messages are processed twice. Exactly-oncecomes from making the second pass harmless: Spark Structured Streaming records its source offsets in a checkpoint and writes to Delta tablesidempotently; Kafka transactions commit output and offsets together. A sink with no such support, such as an email, can still be sent twice.

At-least-once delivery may repeat a message after a crash; exactly-once makes the repeat harmless with an atomic or idempotent write.

Kafka durability settings · Going deeper

Three copies, two in sync, acks=all: the settings that lose nothing

SettingOnDefaultEffectreplication.factortopic1copies of each partition; production topics use 3min.insync.replicastopic1with acks=all, a write is refused when fewer replicas are in syncacksproducerall0: do not wait · 1: leader only · all: every in-sync replicaenable.idempotenceproducertruethe broker drops a retried batch it already hastransactional.idproducernot setwrites to several partitions, and offsets, commit togetherisolation.levelconsumerread_uncommittedread_committed hides messages from aborted transactionsenable.auto.commitconsumertrue, every 5 soffsets commit on a timer, perhaps before the work is doneauto.offset.resetconsumerlatestwhere a new group starts: latest skips all historyunclean.leader.election.enabletopicfalsean out-of-date replica may not become leaderDefaults from the Apache Kafka configuration reference, 3.0 and later (acks was 1 and idempotence off before 3.0).Nothing acknowledged is lost with 3 copies, 2 required in sync, and a producer that waits for all.replication.factor 3 with min.insync.replicas 2 survives one broker failing with no loss and no pause; with two down,producers get NotEnoughReplicas and writes stop, which is the point: refuse, rather than accept a write that one morefailure would lose. On the consumer, turn auto-commit off and commit after processing for at-least-once; a consumer thatcommits first and then crashes has lost the message.

With replication.factor 3, min.insync.replicas 2 and acks=all, an acknowledged message survives a broker failure.

Change data capture

CDC turns every committed change in a database into an event

orders databasetransaction logWAL, binlog, redo logCDC connectorDebezium, DMSevery committed change becomes an event,deletes included{ "op": "u", "before": {"id": 102, "status": "shipped"}, "after": {"id": 102, "status": "delivered"} }{ "op": "d", "before": {"id": 99, "status": "cancelled"}, "after": null }MERGE INTO silver.orders t USING changes s ON t.id = s.id WHEN MATCHED AND s.op = 'd' THEN DELETE WHEN MATCHED THEN UPDATE SET * WHEN NOT MATCHED THEN INSERT *Change data capture reads the database’s own transaction log and turns every insert, update and delete into an event.It adds no queries to the source and misses nothing, unlike polling with a watermark: the delete of order 99 arrives as an event with op "d".Downstream, the events are applied with MERGE to keep a lakehouse copy in step, seconds behind. Debezium (into Kafka), AWS DMS, LakeflowConnect, Fabric mirroring and Delta’s change data feed (changes made to a Delta table) all apply the same idea.

Change data capture reads the database transaction log and emits each insert, update and delete as an event with before and after values.

Applying changes · Going deeper

Apply only the latest change per key, in the source’s order

changes: one micro-batchidopstatussource_ts102udelivered09:55:02102ushipped09:40:10102ushipped09:40:1099dNULL09:50:31103cprocessing09:52:44Order 102 appears three times: its newestchange arrived first, and one was sent twice.Applied in arrival order, 102 ends as shipped.By hand: keep the latest per key, then MERGEMERGE INTO silver.orders t USING ( SELECT * FROM changes QUALIFY ROW_NUMBER() OVER (PARTITION BY id ORDER BY source_ts DESC) = 1) s ON t.id = s.idWHEN MATCHED AND s.op = 'd' THEN DELETEWHEN MATCHED AND s.source_ts > t.source_ts THEN UPDATE SET *WHEN NOT MATCHED AND s.op <> 'd' THEN INSERT *;Declared: Lakeflow Declarative PipelinesAPPLY CHANGES INTO silver.orders FROM STREAM(bronze.changes) KEYS (id) APPLY AS DELETE WHEN op = 'd' SEQUENCE BY source_ts STORED AS SCD TYPE 1;Change events arrive twice and out of order: apply only the latest per key, by a source sequence.Sequence by something the source database assigned: its log position (LSN, binlog offset) or commit time, never the arrivaltime. The check s.source_ts > t.source_ts stops a late, older event overwriting newer data, and one row per key avoids theMERGE error for duplicate matches. APPLY CHANGES (now also written AUTO CDC) does the same from a declaration; SCD TYPE 2keeps history in __START_AT and __END_AT instead of overwriting. Debezium’s op codes: c create, u update, d delete, r a rowread during the first snapshot. The batch above is an example.

Deduplicate by key and order by a source sequence before MERGE; APPLY CHANGES (AUTO CDC) does it from a declaration.

Streaming services

A streaming platform carries, processes, stores and acts on events

JobAzure / FabricAWSOpen source · GoogleEvent streamEvent Hubs, Fabric EventstreamKinesis Data Streams, MSKKafka · Pub/SubDeliver to storageEvent Hubs CaptureData FirehoseKafka ConnectStream processingStream Analytics, DatabricksManaged Service for Apache FlinkFlink, Spark · DataflowReal-time storeEventhouse (KQL database)OpenSearch, TimestreamClickHouse, DruidAct on eventsActivator, Real-Time DashboardsLambda, EventBridge—Capture changesFabric mirroring, DebeziumDMSDebezium · DatastreamA streaming platform has a log of events, an engine that processes them, a store that serves them, and something that acts.Microsoft Fabric Real-Time Intelligence brings these together: Eventstream ingests and routes events, an Eventhouse stores them in KQL databasesfor fast queries over time, Real-Time Dashboards show them and Activator triggers actions when conditions occur. Azure Stream Analytics runsSQL over Event Hubs streams. On AWS, Kinesis or MSK carry events, Flink processes them and Firehose lands them in S3.

Event Hubs, Kinesis and Kafka carry events; Fabric Real-Time Intelligence adds Eventstream, Eventhouse and Activator.

Streaming services · Going deeper

Spark Structured Streaming in practice

A Structured Streaming query reads a source, keeps progress in a checkpoint, and runs on a trigger.

One query, four decisions

  • Source Kafka, Event Hubs, Kinesis, a Delta table, or files through Auto Loader. Sources must be replayable so a restart can re-read from the checkpoint.
  • Checkpoint a folder holding the offsets processed and any state (window counts, deduplication). One per query; never share or delete it, or the query starts over.
  • Trigger the default runs micro-batches back to back; processingTime = "1 minute" runs one a minute; availableNow processes everything waiting, then stops, which gives incremental batch jobs on a schedule.
  • Output mode and sink append writes only new rows (with a watermark for aggregates); update writes changed rows; complete rewrites the whole result. Delta tables are the usual sink, written exactly once.

Stream–static and stream–stream joins

Joining a stream to a static table (orders to the customer dimension) reads the latest table version each micro-batch. Joining two streams needs watermarks on both, so the engine knows how long to remember unmatched rows.

Lesson complete

You finished “Streaming”

Nicely done.

Open in a new tab