A customer taps "Pay" on PhonePe. Their bank balance must update, the merchant must get notified, and a fraud-detection system must approve or block the transaction - all within a few hundred milliseconds. A nightly batch pipeline that processes yesterday's transactions at 2 AM cannot do any of this. The business event has already happened and money has already moved by the time a batch job would even start. This is the gap that streaming systems close. Instead of collecting data and processing it later on a schedule, a streaming system reacts to each event the moment it happens. **Apache Kafka** is one of the most widely used platforms for building this kind of high-throughput event streaming architecture. It is a distributed event streaming platform - a durable, high-throughput log that producers write events into and consumers read events from, independently and in real time. > 💡 **Tip:** Kafka was originally built at LinkedIn to handle the flood of activity data - clicks, views, messages - flowing between dozens of internal systems that all needed the same events without querying each other's databases directly. That original problem, one event needed by many independent consumers, is still exactly what Kafka is best at. ### Message Queues vs Event Streaming Platforms Before Kafka, systems like RabbitMQ or Amazon SQS handled inter-service messaging. Kafka is often lumped in with them, but the two solve different problems. | | Message Queue (RabbitMQ, SQS) | Event Streaming Platform (Kafka) | |---|---|---| | Message lifecycle | Deleted once consumed | Retained for a configured period regardless of consumption | | Replay | Generally not possible | Any consumer can re-read from an earlier offset | | Multiple independent readers | Awkward - typically one consumer per message | Natural - many consumer groups read the same data independently | | Best for | Task distribution, one-time work items | Shared event history, multiple downstream systems | > 📌 **Engineering Decision:** Reach for Kafka when the same event needs to reach multiple independent systems - a payment event that a fraud service, a notification service, and an analytics pipeline all need to see, each at their own pace. Reach for a simpler queue like SQS or RabbitMQ when you have one task that needs to be done exactly once by exactly one worker, like sending a single email. Using Kafka for simple one-consumer task queues is usually more operational overhead than the problem calls for; using a queue for shared event distribution means losing replay and multiple independent readers. ### Batch vs Streaming - When Each Is Right Not every pipeline needs to be real time, and building streaming infrastructure for a problem that batch already solves well is wasted effort. * **Batch** fits a daily sales dashboard, a monthly billing run, or any report where "as of last night" is genuinely good enough. * **Streaming** fits fraud detection, live inventory counts, real-time personalization, or anything where a decision must be made within seconds of the event occurring. ### Concept Check * A retailer wants a dashboard showing yesterday's total revenue by city, refreshed once a day. Would you reach for Kafka here? Why or why not? * Why does a message queue struggle when three completely different systems all need to independently process the same event? ---
### Topics - Named Streams of Events A **topic** is a named category that events are published to - think of it like a named log file that many producers can write to and many consumers can read from. A UPI payments system might have a topic called `payment-events`, and every payment anywhere in the system gets written there. Topics are logical - Kafka does not care what is inside a message. It is just bytes to Kafka. What those bytes mean is a decision made by producers and consumers, not by Kafka itself. ### Partitions - How Topics Scale A topic is split into **partitions** - ordered, independent logs that together make up the full topic. Splitting a topic into partitions is what lets Kafka handle far more throughput than a single machine could manage, because different partitions can be written to and read from in parallel, potentially on different brokers. Topic: payment-events (3 partitions) Partition 0: [msg0] [msg1] [msg2] [msg3] ... Partition 1: [msg0] [msg1] [msg2] ... Partition 2: [msg0] [msg1] [msg2] [msg3] [msg4] ... Order is guaranteed WITHIN a partition, never ACROSS partitions. Each message is assigned to a partition based on its **key**. Every message with the same key always lands in the same partition, which is what gives you ordering guarantees for that key - every event for `merchant_id=RAZORPAY_042` arrives in the exact order it was produced, because they all land in the same partition. > **Note:** More partitions means more consumers can read a topic in parallel, since each partition can only be actively read by one consumer within a given consumer group at a time. But more partitions also means more file handles for the broker to manage and slower consumer group rebalances - partition count is a real capacity-planning decision, not something to maximize blindly. ### Replication - What Happens When a Broker Fails Partitions give you parallelism, but a single copy of a partition sitting on one broker is a single point of failure. **Replication** is how Kafka protects against a broker going down - each partition is copied across multiple brokers, not just stored once. Topic: payment-events Replication factor: 3 Partition 0: Broker 1 -> Leader (producers write here, consumers read here) Broker 2 -> Follower (replicates the leader's data) Broker 3 -> Follower (replicates the leader's data) Every partition has one **leader** - the broker producers actually write to and consumers actually read from - and one or more **followers**, which continuously replicate the leader's data. A follower that is fully caught up with the leader is called an **in-sync replica (ISR)**. If the leader's broker fails, Kafka promotes one of the in-sync replicas to become the new leader automatically, and producers and consumers keep working with no data loss. This is what `min.insync.replicas` and `acks` actually protect. `min.insync.replicas` sets the minimum number of replicas (including the leader) that must be in sync for a write to be allowed at all. Combined with `acks="all"`, this is what determines how durable a write really is - not `acks` alone. > 📌 **Engineering Decision:** Replication is not free - a replication factor of 3 means every byte written is stored three times and replicated over the network three times, which costs both disk and bandwidth. For most production topics carrying data you cannot afford to lose, replication factor 3 with `min.insync.replicas=2` is the standard, battle-tested starting point. Dropping to replication factor 1, as this module's local lab does, is reasonable for local development only - never for anything resembling production. ### Offsets - Your Position in the Log An **offset** is simply the position of a message within its partition - a sequential number starting at 0. A consumer tracks which offset it has read up to, and that number is how Kafka knows "you have already seen everything up through here." ```text Partition 0: offset 0: {"payment_id": "PAY001", "amount": 499} offset 1: {"payment_id": "PAY002", "amount": 1200} offset 2: {"payment_id": "PAY003", "amount": 89} ^ consumer has committed offset 1 - it will resume from offset 2 next ``` ### Kafka's Retention Model - Why It Is Not "Just a Queue" This is the single biggest conceptual difference from a traditional queue. In RabbitMQ, once a consumer acknowledges a message, it is gone. In Kafka, a message stays in its partition for a configured **retention period** - commonly days - regardless of whether anyone has read it yet. This means a brand-new consumer, added to the system three days after an event happened, can still read that event from the beginning if it wants to. It also means the same event can be read independently by five completely different consumer groups, each tracking its own separate offset, without any of them affecting the others. ### Concept Check * Two messages have the same key. Are they guaranteed to arrive in order relative to each other? What about two messages with different keys? * A new analytics team joins your company and wants to backfill three days of `payment-events` history into their warehouse. Why does Kafka's retention model make this straightforward, where a traditional queue would not? ---
### Producers - Writing Events Into Kafka A **producer** is any application that publishes events to a Kafka topic. Producers choose which topic to write to, and optionally which key to attach to each message - the key is what determines the partition, as covered above. ```python from confluent_kafka import Producer ## acks="all" requires the leader to wait until min.insync.replicas replicas have ## acknowledged the write before it is considered successful - durability depends ## on acks together with replication.factor and min.insync.replicas, not acks alone ## enable.idempotence prevents duplicate messages if the producer has to retry a send conf = { "bootstrap.servers": "broker-1:9092,broker-2:9092", "acks": "all", "enable.idempotence": True, } producer = Producer(conf) def delivery_report(err, msg): if err is not None: print(f"Delivery failed: {err}") else: print(f"Delivered to {msg.topic()}[{msg.partition()}] @ offset {msg.offset()}") ## key="merchant_042" ensures every event for this merchant lands in the same partition producer.produce( "payment-events", key="merchant_042", value='{"payment_id": "PAY001", "amount": 499, "status": "success"}', on_delivery=delivery_report, ) producer.flush(5) ## wait up to 5 seconds for all pending messages to be delivered ``` > **Note:** `acks="all"` (equivalent to `acks=-1`) is a durability setting, not a speed one - the producer waits for the leader to confirm the write has reached the number of in-sync replicas set by `min.insync.replicas`. It trades a little latency for a strong guarantee, but the actual durability guarantee comes from `acks`, `replication.factor`, and `min.insync.replicas` working together, not from `acks` in isolation. See the Replication section above for how these settings connect. ### Consumers and Consumer Groups - Parallel Reading A **consumer** reads events from one or more partitions. A **consumer group** is a set of consumers cooperating to read a topic together - Kafka spreads the topic's partitions across the consumers in the group so that each partition is read by exactly one consumer in that group at a time. Topic: payment-events (3 partitions) Consumer Group: fraud-detector Partition 0 ----> Consumer A Partition 1 ----> Consumer B Partition 2 ----> Consumer C Add a 4th consumer with only 3 partitions? It sits idle - a partition can only be read by ONE consumer per group. ```python from confluent_kafka import Consumer conf = { "bootstrap.servers": "broker-1:9092,broker-2:9092", "group.id": "fraud-detector", "auto.offset.reset": "earliest", ## if no committed offset exists, start from the beginning "enable.auto.commit": False, ## commit manually, only after processing succeeds } consumer = Consumer(conf) consumer.subscribe(["payment-events"]) while True: msg = consumer.poll(1.0) if msg is None: continue if msg.error(): print(f"Consumer error: {msg.error()}") continue process_payment(msg.value()) ## your actual business logic consumer.commit(msg) ## only commit after processing succeeds ``` > 🔴 **Common Mistake:** Leaving `enable.auto.commit` at its default of `True` for anything where losing a message matters. Auto-commit advances the offset on a timer, regardless of whether your code actually finished processing that message - if your consumer crashes mid-processing, the offset may already have advanced past a message you never actually handled, and it is gone from this consumer's perspective. Commit manually, after processing succeeds, whenever correctness matters more than raw simplicity. ### Multiple Independent Consumer Groups Two different consumer groups reading the same topic never interfere with each other - each group tracks its own offsets entirely independently. Topic: payment-events Consumer Group "fraud-detector" ----> reads independently, its own offsets Consumer Group "warehouse-sink" ----> reads independently, its own offsets Both groups can read the SAME events at completely different paces without coordinating or blocking each other in any way. > 📌 **Engineering Decision:** When multiple systems need the same event stream - a fraud service, a warehouse loader, a real-time dashboard - give each one its own consumer group rather than trying to fan a single consumer's output out to multiple downstream systems in application code. Separate consumer groups is what Kafka is built for; it costs nothing extra and each system can fail, restart, or fall behind independently without affecting the others. ### Consumer Rebalancing A **rebalance** is what Kafka does whenever the set of consumers in a group changes - a consumer joins, a consumer leaves cleanly, or a consumer crashes. Kafka redistributes the group's partitions across whichever consumers remain, so every partition still has exactly one owner in the group. Before: Consumer B crashes Consumer A -> Partition 0 After rebalance: Consumer B -> Partition 1 -------------------> Consumer A -> Partition 0, 1 Consumer C -> Partition 2 Consumer C -> Partition 2 During a rebalance, consumption from the affected partitions briefly pauses while ownership is reassigned - this is normal, expected behavior, not a bug. If you ever see log lines mentioning `rebalance`, `partition revoked`, or `partition assigned`, this is exactly what is happening: the group is adjusting to a consumer joining or leaving. > **Note:** Frequent, repeated rebalancing is a real operational problem, though - it usually means consumers are crashing, restarting, or being killed by orchestration tooling too often. A healthy consumer group rebalances rarely, typically only during deliberate scaling or deployment events. ### Concept Check * A consumer group has 3 consumers and the topic has 3 partitions. You add a 4th consumer to the group. What happens to it? * Why is committing an offset only after processing succeeds safer than auto-committing on a timer? > 💡 **Practice:** Using the producer and consumer code above as a starting point, write a small Python producer that sends 20 simulated order events with a shared key for the same `restaurant_id`, then write a consumer that prints each event along with its partition and offset. Confirm every event for that key lands in the same partition. ---
Writing a custom producer or consumer for every single system you need to connect to Kafka does not scale - you would be maintaining dozens of small, similar programs. **Kafka Connect** is a framework specifically for moving data in and out of Kafka using pre-built, configuration-driven connectors instead of custom code. ### Source and Sink Connectors A **source connector** reads from an external system and writes into a Kafka topic. A **sink connector** reads from a Kafka topic and writes into an external system. Source Connector Sink Connector (external system -> Kafka) (Kafka -> external system) PostgreSQL ----> [Debezium] ----> Kafka topic ----> [S3 Sink] ----> S3 bucket ### Standalone vs Distributed Mode Kafka Connect runs as one or more **worker** processes. In **standalone mode**, a single worker runs with no fault tolerance - fine for local development, not for production. In **distributed mode**, multiple workers form a group, and Connect automatically spreads connector tasks across them - if one worker dies, its tasks are reassigned to the survivors. > 📌 **Engineering Decision:** Use Kafka Connect instead of hand-writing a producer or consumer whenever a pre-built connector already exists for the system you are integrating with - a JDBC source, an S3 sink, a Debezium CDC source. Kafka Connect is excellent for reliably moving data between systems, but it is not the right tool once the pipeline needs complex enrichment, joins across streams, branching logic, or stateful processing - that is a job for a stream-processing framework or an application-level consumer instead. Not every Kafka problem is a Kafka Connect problem; it solves data movement specifically, not business logic. ### Concept Check * You need to continuously stream every row change from a production PostgreSQL database into Kafka. Would you write a custom Python producer, or reach for Kafka Connect? Why? * What is the practical risk of running Kafka Connect in standalone mode for a production pipeline? ---
### The Problem with Query-Based Extraction A common but flawed approach to getting data out of a database and into Kafka is to periodically query it - "give me every row where `updated_at > last_run_time`." This has real problems: it misses deletes entirely, it puts repeated load on the source database, and if a row changes twice between polls, you only ever see the final state, never the intermediate change. ### How CDC and Debezium Actually Work **Change Data Capture (CDC)** solves this by reading the database's own internal transaction log - the same log the database uses internally to guarantee durability - instead of querying tables directly. Every insert, update, and delete that the database commits is captured as a change event, in the order it was committed to that log, with zero extra load placed on the tables themselves. **Debezium** is the standard open-source CDC tool, and it ships as a Kafka Connect source connector. PostgreSQL write-ahead log (WAL) | v Debezium connector (reads the WAL, not the tables) | v Kafka topic: orders.public.orders | {"before": {...}, "after": {...}, "op": "u", "ts_ms": ...} > **Note:** Debezium emits one JSON (or Avro) message per row change, containing both the `before` and `after` state of the row and an `op` field - `c` for create, `u` for update, `d` for delete. Downstream consumers get the full picture of exactly what changed, not just the final row state. Ordering is guaranteed relative to the source database's own log and the Kafka partitioning strategy used - once events for different rows land in different partitions, there is no single global order across the whole topic, only ordering within each partition, exactly as covered in the Partitions section above. ### Concept Check * Why does query-based polling for changes miss deletes entirely, while CDC does not? * Two updates happen to the same row within one second, between two polling cycles of a query-based approach. What does the query-based approach see? What does CDC see? ---
### The Problem Schemas Solve Kafka does not look inside your messages - to Kafka, every message is just an opaque array of bytes. This is flexible, but it means nothing stops a producer from silently changing the shape of its data. If a producer renames a field or removes one, every consumer expecting the old shape breaks - often with no clear error until something downstream quietly starts failing. > 💡 **Tip:** In production, every Kafka topic functions like an API contract between whoever produces to it and everyone who consumes from it. Treat a breaking schema change on a shared topic with the same seriousness as a breaking change to a public API endpoint. ### How Schema Registry Works **Schema Registry** is a separate service that stores every version of every schema used on your topics. Producers register a schema before publishing; consumers fetch the matching schema to correctly decode messages. Critically, Schema Registry can enforce **compatibility rules**, rejecting a schema change that would break existing consumers before it ever reaches production. * **Backward compatible** - new schema can read data written with the old schema * **Forward compatible** - old schema can read data written with the new schema * **Full compatible** - both directions hold at once **Avro** is a common schema format in Kafka ecosystems, alongside Protobuf and JSON Schema - each with different trade-offs. | Format | Trade-off | |---|---| | Avro | Compact binary, strong schema evolution support, deep Kafka ecosystem integration | | Protobuf | Common in application/service ecosystems, strongly typed, cross-language | | JSON Schema | Easiest to read and debug, natural fit for JSON-centric systems, more overhead on the wire | ```text ## Registering a schema (conceptually - via the Registry's REST API) POST /subjects/payment-events-value/versions { "schema": "{\"type\":\"record\",\"name\":\"Payment\",\"fields\":[{\"name\":\"payment_id\",\"type\":\"string\"},{\"name\":\"amount\",\"type\":\"double\"}]}" } ``` > 🔴 **Common Mistake:** Treating Schema Registry as optional in the early stages of a project and only adding it once something has already broken. Schema drift causes silent consumer failures - a producer's small, well-intentioned change quietly breaks a downstream consumer that nobody was watching, and it is often discovered only when a report looks wrong days later. Enforcing schema compatibility in CI, before code reaches production, is what prevents this - not monitoring after the fact. ### Concept Check * A producer wants to add a new required field with no default value to an existing topic's schema. Would this pass a backward-compatibility check? Why or why not? * Why is a schema change on a shared Kafka topic comparable to a breaking change on a public API? ---
A customer taps "Pay" on PhonePe. Their bank balance must update, the merchant must get notified, and a fraud-detection ...
Topics - Named Streams of Events A topic is a named category that events are published to - think of it like a named log...
Producers - Writing Events Into Kafka A producer is any application that publishes events to a Kafka topic. Producers ch...
Writing a custom producer or consumer for every single system you need to connect to Kafka does not scale - you would be...
The Problem with Query-Based Extraction A common but flawed approach to getting data out of a database and into Kafka is...
The Problem Schemas Solve Kafka does not look inside your messages - to Kafka, every message is just an opaque array of ...
Retention by Age vs Retention by Key Standard Kafka retention deletes messages after a configured time period regardless...
Delivery Semantics At-least-once - a message may be delivered more than once (for example, after a producer retry) but i...
This lab has two parts. Part 1 builds a Kafka event pipeline from scratch and checks consumer lag. Part 2 adds a real CD...
Term / Command What it Means Topic Named stream of events Partition Ordered sub-log of a topic; unit of parallelism Repl...
Aligns directly with DevOps, Site Reliability (SRE), and Platform Engineering job descriptions.