Master core concepts and production patterns.
### The Problem Kafka Solves Picture a growing platform — an order service, an inventory service, a notification service, and a logging service all need to communicate with each other. The naive approach is direct connections: Order calls Inventory, Inventory calls Notification, and so on. Direct connections — what breaks: Order Service ──► Inventory Service │ │ └──► Notification └──► Logging └──► Analytics If Inventory is slow → Order waits If Notification crashes → Order fails If traffic spikes → everything falls over Every service is tightly dependent on every other service. One failure cascades into a full outage. **The two core problems are:** * How do you move large volumes of data between services reliably and quickly? * What happens to messages when a service goes down — are they lost? ### What is a Messaging System A messaging system sits between services and handles communication on their behalf. A service drops a message into the system and moves on — it does not wait. Another service picks that message up when it is ready. The two services never talk directly. Two patterns exist in messaging systems: | Pattern | How it Works | Key Behaviour | |---|---|---| | Point-to-point | Message goes into a queue, one consumer reads it and it's removed | Each message consumed once | | Publish-subscribe | Message goes into a topic, every subscriber gets a copy | Many consumers, same message | Kafka follows the **publish-subscribe** pattern, but it is also flexible enough to behave like a queue through consumer groups, which you will see in Topic 07. ### What is Kafka Apache Kafka is a distributed, fault-tolerant, high-throughput data streaming platform. It was built at LinkedIn to handle their massive activity data and open-sourced in 2011. Today it is one of the most widely used data infrastructure tools in the world. At its core, Kafka is a **distributed log**. When a service sends a message, Kafka writes it to disk and keeps it for a configurable period. Unlike a traditional queue where a message disappears after it is read, Kafka keeps messages around. Consumers can re-read old messages, catch up after downtime, or replay historical data entirely. How Kafka fits in: Service A (Producer) ──► Kafka ──► Service B (Consumer) └──► Service C (Consumer) └──► Service D (Consumer) Service A sends once. Every consumer gets the message independently. If Service C is down, it catches up when it comes back — no data lost. ### Why Kafka is Faster Than Traditional Systems | Problem | Traditional Queue | Kafka | |---|---|---| | Multiple consumers | One message reaches one consumer | All consumer groups get every message independently | | Consumer is slow | Queue backs up, blocks everything | Consumer reads at its own pace, no blocking | | Service crashes | Messages in flight may be lost | Messages replicated across brokers, nothing lost | | Replay old data | Not possible once consumed | Possible — reset the offset and re-read | | High traffic | Performance degrades quickly | Built for millions of messages per second | Kafka achieves this speed through four design decisions: **Sequential disk writes** — Kafka only appends new data to the end of a log file. It never modifies existing data. Sequential writes on modern disks are extremely fast, nearly as fast as writing to memory. **Zero-copy transfer** — Instead of reading data from disk into memory and then copying it to the network, Kafka tells the operating system to move data from disk directly to the network socket. This removes multiple unnecessary copy steps. **Batching** — Producers group many messages into one batch before sending. Fewer network trips means far higher throughput. **Partitioning** — Topics are split into partitions spread across many machines. More machines means more parallel throughput, and you can always add more. ### Real-World Use Cases | Company | How They Use Kafka | |---|---| | LinkedIn | Activity feeds, newsfeed ranking, metrics | | Netflix | Real-time monitoring, error alerting, event processing | | Uber | Location tracking, ride events, surge pricing calculations | | Banks | Fraud detection, real-time transaction alerts | | Zomato/Swiggy | Order events, delivery tracking, notifications | General categories where Kafka is used: * **Activity tracking** — every user action (click, login, purchase) sent as an event * **Log aggregation** — collecting logs from many services into one central stream * **Metrics pipelines** — streaming stats from distributed systems into dashboards * **Real-time data pipelines** — moving data between systems reliably at scale * **Event-driven microservices** — services that react to events rather than calling each other ---
### Broker A **broker** is a single Kafka server process. It receives messages from producers, writes them to disk, and serves them to consumers. Every broker in a cluster can handle hundreds of thousands of reads and writes per second and store terabytes of data without slowing down. One important thing: brokers are **stateless** — they do not track what each consumer has read. That responsibility belongs to the consumer itself. When you start Kafka, you are starting a broker. In production, you run multiple brokers together as a cluster. ### Topic A **topic** is a named category where messages are stored. Think of it as a named stream of data. Producers write to a topic, consumers read from a topic. Examples of topic names in real systems: ```bash ## Topic names from a real platform orders payments user-logins ride-location-updates inventory-alerts ``` Topics are not limited to one machine. They are split into partitions distributed across multiple brokers, which is what makes them scalable. ### Log Internally, every topic partition is stored as a **log** — an append-only file on disk. New messages are always added to the end. Existing messages are never overwritten or modified. Log structure (append-only): [msg 0] [msg 1] [msg 2] [msg 3] [msg 4] ← new messages go here This is why Kafka calls itself a distributed log — the log is the fundamental unit of storage. ### Partition A **partition** is a subdivision of a topic. Instead of storing all messages for a topic in one log on one machine, the topic is split into N partitions, each stored on a different broker. Topic: ride-location-updates (3 partitions) Partition 0: [msg0] [msg3] [msg6] ... → Broker 1 Partition 1: [msg1] [msg4] [msg7] ... → Broker 2 Partition 2: [msg2] [msg5] [msg8] ... → Broker 3 Partitions give Kafka two critical properties: * **Parallelism** — multiple consumers can read from different partitions at the same time * **Scalability** — data is spread across machines so you can handle far more load | | Log | Partition | |---|---|---| | What it is | Physical file on disk | Logical division of a topic | | Visible on disk | Yes — you can see the files | No — managed by Kafka internally | | Purpose | Where data is actually stored | How topics scale across machines | ### Offset Every message written to a partition gets a unique sequential integer called an **offset**. Offsets start at 0 and increase by one for every new message. Partition 0: [offset 0] [offset 1] [offset 2] [offset 3] [offset 4] Consumers use offsets to track where they are in the log. If a consumer crashes and restarts, it reads its last committed offset and picks up exactly where it left off. No messages are skipped, no messages are read twice. ### Message Structure Every message Kafka stores is called a **record**. Each record contains: | Field | Description | |---|---| | Topic | Which topic this record belongs to | | Partition | Which partition it was written to | | Offset | Its position within that partition | | Key | Optional — used to route to a specific partition | | Value | The actual payload — stored as raw bytes | | Timestamp | When the message was produced | ### Producer A **producer** is any application or service that writes messages to a Kafka topic. Producers send messages as fast as the broker can accept them. By default, messages are spread evenly across all partitions. If a key is included with the message, Kafka uses the key to decide which partition gets it — guaranteeing that all messages with the same key always land in the same partition and stay in order. ### Consumer A **consumer** is any application or service that reads messages from a Kafka topic. Consumers **pull** data from brokers — they request messages themselves rather than having Kafka push messages to them. Each consumer keeps track of its own offset, so it decides exactly where in the log to read from and at what pace. ### Leader and Follower Every partition has one **leader** and one or more **followers** (also called replicas): * All reads and writes for a partition go through the **leader** * **Followers** silently copy data from the leader and stay in sync * If the leader broker crashes, one of the followers is automatically promoted as the new leader Partition 0 with replication factor 3: Leader → Broker 1 ← all reads and writes go here Follower → Broker 2 (copying from leader silently) Follower → Broker 3 (copying from leader silently) If Broker 1 goes down → Broker 2 or 3 becomes the new leader automatically > 📌 **Remember:** A topic is the category. A partition is how that category is split for scale. An offset is how a consumer tracks its position. These three concepts are the foundation everything else builds on. ---
### How a Kafka Cluster Works A **cluster** is a group of brokers working together. Data is distributed and replicated across them. If one broker fails, the others already have copies of the data and keep serving producers and consumers without interruption. Kafka Cluster — 3 brokers, 1 topic, 3 partitions, replication factor 3: Broker 1: P0-leader, P1-follower, P2-follower Broker 2: P0-follower, P1-leader, P2-follower Broker 3: P0-follower, P1-follower, P2-leader Every broker holds some partition leaders and some followers. The load is shared evenly across the cluster. When a client connects to any one broker, that broker automatically returns the addresses of all other brokers in the cluster. The client only needs to know one address to discover the whole cluster — this is called the **bootstrap server**. ### Cluster Controller One broker in the cluster is elected as the **cluster controller**. In addition to its normal broker duties, the controller: * Assigns partitions to brokers when new topics are created * Monitors for broker failures * Triggers leader election when a broker goes down Only one controller exists at any time. If it fails, the remaining brokers elect a new one automatically. ### Replication Factor and ISR **Replication factor** is how many copies of each partition exist across the cluster. | Replication Factor | Meaning | Brokers That Can Fail Safely | |---|---|---| | 1 | No replication | 0 — any failure means data loss | | 2 | One copy | 1 | | 3 | Two copies | 2 | > 📌 **Remember:** Always use replication factor 3 in production. It lets two brokers go down without losing any data or service. **ISR — In-Sync Replicas** is the set of follower replicas that are fully caught up with the leader. A follower is in the ISR only if it has replicated all messages the leader currently has. Partition 0, replication factor 3: Leader: Broker 1 ✓ (always in ISR) Follower: Broker 2 ✓ (fully caught up — in ISR) Follower: Broker 3 ✗ (behind — temporarily out of ISR) ISR = [Broker 1, Broker 2] Only ISR members can become the new leader on failover. This guarantees no data loss — a replica that is behind cannot become the leader and serve stale data. ### What Happens When a Broker Goes Down ```bash ## Scenario: Broker 2 goes down ## Step 1 — Controller detects the failure ## Step 2 — For every partition where Broker 2 was the leader, ## controller picks a new leader from the ISR ## Step 3 — Producers and consumers are redirected to the new leader ## Step 4 — When Broker 2 comes back online, ## it catches up as a follower before rejoining the ISR ``` From a producer and consumer perspective, this failover is automatic and typically takes only a few seconds. ### ZooKeeper — The Old Way (Deprecated) > 📌 **Remember:** ZooKeeper is kept here so you understand what it did and why KRaft was built to replace it. You will still see ZooKeeper mentioned in older documentation and job descriptions, so knowing it matters. Originally, Kafka required a separate **ZooKeeper** cluster running alongside it. ZooKeeper was a distributed coordination service that handled: * Tracking which brokers were alive * Storing topic and partition metadata * Managing leader elections between brokers * Tracking consumer group offsets (in older versions) This meant running two completely separate systems — Kafka and ZooKeeper — and keeping both healthy. ZooKeeper also had a hard limit on how many partitions a cluster could support, which became a problem as clusters grew. Old architecture (ZooKeeper mode): ZooKeeper Cluster ← all metadata stored here ↓ Kafka Cluster ← brokers check ZooKeeper for everything Problems: * Two systems to deploy and maintain * ZooKeeper partition limit: ~200,000 * Slower startup — Kafka had to wait for ZooKeeper * Failure of ZooKeeper could affect the entire Kafka cluster ### KRaft — The Modern Way From Kafka 3.x, **KRaft** (Kafka Raft) replaced ZooKeeper entirely. In KRaft mode, Kafka manages its own metadata internally using the Raft consensus algorithm — the same algorithm used in many modern distributed systems. Some brokers are designated as **controllers**. They use Raft to elect a leader among themselves and store all cluster metadata in an internal Kafka topic called `__cluster_metadata`. No external system needed. KRaft architecture: Broker + Controller 1 ← metadata leader (elected) Broker + Controller 2 ← follower (ready to take over) Broker + Controller 3 ← follower All metadata lives inside Kafka itself — no ZooKeeper needed | | ZooKeeper Mode | KRaft Mode | |---|---|---| | External dependency | Yes — ZooKeeper cluster required | No | | Partition limit | ~200,000 | Millions | | Startup | Slow — waits for ZooKeeper | Fast | | Maintenance overhead | Two systems | One system | | Production status | Deprecated — do not use for new setups | Recommended for all new clusters | > 🔴 **Common Mistake:** Setting up new Kafka clusters with ZooKeeper mode. Always use KRaft for any new installation. ---
### How Partitions Increase Throughput Without partitions, a topic is a single log on one server. Every producer write and every consumer read competes for that one file on that one machine. With partitions, the load is distributed: Without partitions: With 3 partitions: Producer → [single log] Producer → Partition 0 → Broker 1 Consumer ← [single log] Producer → Partition 1 → Broker 2 Producer → Partition 2 → Broker 3 3 parallel writes, 3 parallel reads More partitions means more parallelism. More parallelism means higher throughput. However, more partitions also means more file handles and more leader elections to manage, so do not create unnecessarily large numbers of partitions. ### Partition Count Guidelines | Cluster Size | Recommended Partitions Per Topic | |---|---| | Small — fewer than 6 brokers | 2 × number of brokers | | Large — more than 12 brokers | 1 × number of brokers | Start conservatively and increase if needed. You can always add more partitions but you can never reduce them. ### Key-Based Partition Assignment When a producer sends a message **with a key**, Kafka determines the partition using: ```bash ## How Kafka assigns partitions when a key is present target_partition = hash(key) % number_of_partitions ``` Same key always produces the same hash, which means the same partition. This guarantees that all messages sharing the same key land in the same partition and are read in order. Example: all ride events for driver `driver-7821` always go to the same partition, so the consumer always sees those events in the correct sequence. When a producer sends **without a key**, Kafka distributes messages across partitions in round-robin — maximises throughput but provides no ordering guarantee across partitions. ### Offset Tracking and Message Replay Kafka stores committed consumer offsets in an internal topic called `__consumer_offsets`. When a consumer commits an offset, it records: "I have successfully processed everything up to this point." Partition 0: [0] [1] [2] [3] [4] [5] ↑ consumer committed offset = 3 next read will start from offset 4 A powerful feature of Kafka that traditional queues cannot offer: consumers can **rewind** their offset to re-read old messages. This is useful for replaying events after a bug fix or reprocessing historical data without any changes to the producer side. ### Ordering Guarantees Kafka guarantees message order **within a partition** — messages are always delivered in the exact order they were written. There is no ordering guarantee **across** different partitions. Partition 0: event-A → event-B → event-C (order guaranteed) Partition 1: event-X → event-Y → event-Z (order guaranteed) A consumer reading both partitions may see: A, X, B, Y, C, Z The order across partitions is not guaranteed **Rule:** If ordering matters for a set of related messages, always use the same key so they always land in the same partition. > 📌 **Remember:** Key → same partition → ordering guaranteed. No key → round-robin → no ordering guarantee across partitions. ---
### The Full Flow of Sending a Message When a producer sends a message, this is exactly what happens step by step: 1. Producer creates a record (topic, value, optional key) ↓ 2. Key and value are serialised to bytes ↓ 3. Partitioner decides which partition: - If partition specified directly → use it - If key provided → hash(key) % total partitions - If no key → round-robin across partitions ↓ 4. Record is added to an in-memory batch for that partition ↓ 5. Batch is sent to the broker when full or linger.ms is reached ↓ 6. Broker responds: success (with offset) or error ↓ 7. On error → producer retries if retries > 0 ### acks — Acknowledgement Settings `acks` controls when the producer considers a write successful: | acks Value | Meaning | Risk | Speed | |---|---|---|---| | `0` | Do not wait for any acknowledgement | Data loss possible | Fastest | | `1` | Wait for the leader broker to confirm the write | Lost if leader crashes before replication | Fast | | `all` | Wait for all ISR replicas to confirm | No data loss | Slowest | ```python ## Producer with acks=all — safest setting for critical data from kafka import KafkaProducer producer = KafkaProducer( bootstrap_servers='localhost:9092', acks='all' ) ``` `acks=all` must be used together with `min.insync.replicas`. If replication factor is 3 and `min.insync.replicas=2`, at least 2 brokers must confirm the write before the producer gets a success response. ```bash ## What this means in practice: ## replication.factor = 3 ## min.insync.replicas = 2 ## acks = all ## → At least 2 of 3 brokers must have the message before success is returned ## → Can tolerate 1 broker going down with zero data loss ## → If only 1 broker is alive, producer gets NotEnoughReplicasException ``` ### Idempotent Producer **The problem:** when a network error causes the producer to retry, the broker may have already written the message. The retry creates a duplicate. Producer sends msg → broker writes it → network drops → producer retries → duplicate! **The solution:** an idempotent producer. Kafka assigns each producer session a unique ID and tracks sequence numbers per partition. If a retry arrives with a sequence number the broker has already seen, the duplicate is silently dropped. ```python ## Enable idempotent producer in Python from kafka import KafkaProducer producer = KafkaProducer( bootstrap_servers='localhost:9092', enable_idempotence=True ## handles duplicates from retries automatically ) ``` ### Batching and linger.ms By default, Kafka tries to send records immediately. Increasing `linger.ms` tells the producer to wait a short time for more messages to arrive before sending, forming a larger batch. ```python from kafka import KafkaProducer import json producer = KafkaProducer( bootstrap_servers='localhost:9092', value_serializer=lambda v: json.dumps(v).encode('utf-8'), batch_size=32768, ## 32KB batch size (default is 16KB) linger_ms=10 ## wait 10ms for more messages before sending ) ``` Larger batches mean fewer network requests, which means higher throughput. The tradeoff is a small increase in latency. ### Compression Compression reduces the size of batches before they go over the network: ```python producer = KafkaProducer( bootstrap_servers='localhost:9092', compression_type='snappy' ## options: 'snappy', 'lz4', 'gzip', 'zstd' ) ``` | Format | Speed | Compression | Best For | |---|---|---|---| | none | Fastest | None | Development only | | snappy | Fast | Good | General production use | | lz4 | Very fast | Good | High-throughput pipelines | | gzip | Slow | Best ratio | When bandwidth is the bottleneck | Compression is applied at the batch level. Bigger batches compress much better than small ones — another reason to increase `linger.ms` in high-throughput setups. ### Producer Configuration Reference | Config | Default | What it Does | |---|---|---| | `bootstrap_servers` | — | Comma-separated list of broker addresses. Include at least 2 for resilience | | `acks` | `1` | Acknowledgement level — `0`, `1`, or `all` | | `retries` | `2147483647` | How many times to retry on failure | | `batch_size` | `16384` (16KB) | Max batch size per partition in bytes | | `linger_ms` | `0` | How long to wait for more messages before sending | | `buffer_memory` | `33554432` (32MB) | Total memory for buffering messages before sending | | `compression_type` | `none` | Compression — `snappy`, `lz4`, `gzip`, `zstd` | | `enable_idempotence` | `False` | Prevent duplicates from retries | > 💡 **Tip:** For production systems handling critical data use `acks=all`, `enable_idempotence=True`, and `compression_type='snappy'` as your starting configuration. Tune `linger_ms` and `batch_size` based on your throughput requirements. ---
Master this concept and view production exercises.
The Problem Kafka Solves Picture a growing platform — an order service, an inventory service, a notification service, an...
Broker A broker is a single Kafka server process. It receives messages from producers, writes them to disk, and serves t...
How a Kafka Cluster Works A cluster is a group of brokers working together. Data is distributed and replicated across th...
How Partitions Increase Throughput Without partitions, a topic is a single log on one server. Every producer write and e...
The Full Flow of Sending a Message When a producer sends a message, this is exactly what happens step by step: Producer ...
The Pull Model Consumers pull data from brokers — they request messages in a loop rather than having Kafka push messages...
The Problem One Consumer Cannot Solve If a single consumer reads a topic with 6 partitions, it processes all 6 partition...
Why Kafka Only Stores Bytes Kafka is completely language-agnostic. It does not understand Python dictionaries, JSON obje...
Prerequisites — Java Kafka runs on the JVM. Verify Java is installed first: If Java is not installed: Kafka requires Jav...
Topics Console Producer Console Consumer Consumer Groups The LAG column is the most important metric to monitor. It show...
Setup Complete Producer Example Complete Consumer Example with Manual Commit Producer Configuration Reference Config Def...
Retention Policies Kafka does not delete messages after they are read. Messages are kept based on a configured retention...
The Kafka Integration Pattern Kafka acts as the central hub in most data pipelines. Other systems connect as producers o...
> 📌 Remember: This lab is the most important section in the module. You will build a real working system — four service...
Apache Kafka — Interview Questions & Answers > 📌 Remember: These questions cover beginner to advanced level. Work throu...
Aligns directly with DevOps, Site Reliability (SRE), and Platform Engineering job descriptions.