Deploy a Production Kafka Event Streaming Platform on Kubernetes

Deploy a 3-broker Kafka cluster with Strimzi operator, build producer and consumer services, configure Kafka Connect PostgreSQL sink, and monitor with Grafana.

Related Concepts & TermsProcess Deployment grep Grafana ConfigMap Artifact

Domains & Technologies

Domains

KAFKAMONITORING

Technologies

KUBERNETESTERRAFORM

Blueprint Walkthrough

Architecture Overview

This project deploys a production-grade Apache Kafka event streaming platform on Kubernetes using the Strimzi operator. Kafka is the backbone of event-driven architectures — at Zerodha, every trade placed by every user flows through Kafka before being processed by the matching engine, risk system, and ledger simultaneously.

You will build the full event streaming pipeline: a 3-broker Kafka cluster with replication, a producer service that publishes order events, a consumer service that processes them, Kafka Connect to automatically sink processed events into PostgreSQL, and Grafana dashboards to monitor broker health, consumer lag, and throughput.

◈ DIAGRAM

    Order Service
    (producer)
            |
            v
      +-----------+
      | Kafka     |  Broker 0 (kafka-0)
      | Cluster   |  Broker 1 (kafka-1)  <- 3 brokers, replication factor 3
      | (Strimzi) |  Broker 2 (kafka-2)
      +-----------+
            |
      +-----+-----+
      |           |
      v           v
Consumer      Kafka Connect
Service       (sink connector)
(processes    (writes to PostgreSQL
 orders)       automatically)
      |           |
      v           v
Analytics     PostgreSQL
Dashboard     Database

Problem Solved

Without an event streaming platform, microservices communicate directly — the order service calls the payment service, the notification service, and the analytics service synchronously. If any downstream service is slow or down, the order service hangs. This is tight coupling and it is fragile at scale.

With Kafka, the order service publishes an event and its job is done. The payment service, notification service, and analytics service each consume the event independently. They can be down for hours, catch up when they come back, and the order service never knew anything was wrong. This is how Zerodha processes 10 million orders per day without the trading API slowing down during peak hours.

Step-by-Step Implementation Guide

Step 1: Install Strimzi Kafka Operator

Strimzi is a Kubernetes operator that manages Kafka as a native Kubernetes workload. Instead of manually managing Kafka brokers, you declare what you want in a YAML file and Strimzi handles the rest — rolling upgrades, configuration changes, scaling.

Bash

## Create the Kafka namespace
kubectl create namespace kafka
 
## Install Strimzi operator
## The operator watches for Kafka custom resources and manages the cluster
kubectl create -f 'https://strimzi.io/install/latest?namespace=kafka' \
  -n kafka
 
## Wait for the Strimzi operator to be ready
kubectl get pods -n kafka --watch
## Expected: strimzi-cluster-operator-xxx Running
 
## Verify the Kafka CRDs were installed
kubectl get crd | grep kafka
## Expected: kafkas.kafka.strimzi.io, kafkatopics.kafka.strimzi.io,
##           kafkausers.kafka.strimzi.io, kafkaconnects.kafka.strimzi.io

Remember
The Strimzi operator is the manager. It does not run Kafka itself — it watches for Kafka custom resources and creates the actual Kafka pods. This is the Kubernetes operator pattern — you declare intent, the operator makes it happen.

Step 2: Deploy a Production 3-Broker Kafka Cluster

Bash

## Create the Kafka cluster custom resource
kubectl apply -n kafka -f - <<EOF
apiVersion: kafka.strimzi.io/v1beta2
kind: Kafka
metadata:
  name: production-cluster
spec:
  kafka:
    version: 3.7.0
    replicas: 3  # 3 brokers for high availability
 
    listeners:
      * name: plain
        port: 9092
        type: internal
        tls: false
      * name: tls
        port: 9093
        type: internal
        tls: true
 
    config:
      # Replication and durability settings
      offsets.topic.replication.factor: 3
      transaction.state.log.replication.factor: 3
      transaction.state.log.min.isr: 2  # Minimum in-sync replicas
      default.replication.factor: 3
      min.insync.replicas: 2
 
      # Performance tuning
      num.network.threads: 3
      num.io.threads: 8
      socket.send.buffer.bytes: 102400
      socket.receive.buffer.bytes: 102400
      socket.request.max.bytes: 104857600
 
      # Log retention
      log.retention.hours: 168  # Keep messages for 7 days
      log.retention.bytes: 107374182400  # 100GB per partition max
      log.segment.bytes: 1073741824  # 1GB segments
 
    storage:
      type: jbod
      volumes:
      * id: 0
        type: persistent-claim
        size: 100Gi
        deleteClaim: false
 
    resources:
      requests:
        memory: 2Gi
        cpu: 500m
      limits:
        memory: 4Gi
        cpu: 2
 
    metricsConfig:
      type: jmxPrometheusExporter
      valueFrom:
        configMapKeyRef:
          name: kafka-metrics
          key: kafka-metrics-config.yml
 
  zookeeper:
    replicas: 3  # Zookeeper ensemble for cluster coordination
    storage:
      type: persistent-claim
      size: 10Gi
      deleteClaim: false
    resources:
      requests:
        memory: 1Gi
        cpu: 250m
 
  entityOperator:
    topicOperator: {}  # Manages KafkaTopic resources
    userOperator: {}   # Manages KafkaUser resources
EOF
 
## This takes 3-5 minutes for all brokers to start
kubectl get kafka -n kafka --watch
## Expected: production-cluster shows READY=True
 
## Verify all 3 brokers are running
kubectl get pods -n kafka -l strimzi.io/name=production-cluster-kafka
## Expected: production-cluster-kafka-0, -1, -2 all Running 1/1

Step 3: Create Topics with Proper Configuration

Bash

## Create the orders topic using Strimzi's KafkaTopic custom resource
## This is better than using kafka-topics.sh because:
## 1. It is declarative and version-controlled
## 2. Strimzi reconciles the topic if it drifts
## 3. Config changes can be applied via kubectl
 
kubectl apply -n kafka -f - <<EOF
apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaTopic
metadata:
  name: orders
  labels:
    strimzi.io/cluster: production-cluster  # Must match the Kafka cluster name
spec:
  partitions: 6   # 6 partitions for parallelism (2 per broker)
  replicas: 3     # Each partition replicated on all 3 brokers
  config:
    retention.ms: 604800000     # 7 days in milliseconds
    cleanup.policy: delete      # Delete old messages (not compact)
    min.insync.replicas: "2"    # Producer must get ack from at least 2 replicas
    compression.type: lz4       # Compress messages for storage efficiency
---
apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaTopic
metadata:
  name: payments
  labels:
    strimzi.io/cluster: production-cluster
spec:
  partitions: 6
  replicas: 3
  config:
    retention.ms: 2592000000  # 30 days — payment events kept longer
    min.insync.replicas: "2"
---
apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaTopic
metadata:
  name: notifications
  labels:
    strimzi.io/cluster: production-cluster
spec:
  partitions: 3   # Lower volume — fewer partitions
  replicas: 3
  config:
    retention.ms: 86400000  # 1 day — notification events expire quickly
EOF
 
## Verify topics were created
kubectl get kafkatopics -n kafka
## Expected: orders, payments, notifications all showing READY=True, PARTITIONS, REPLICATION FACTOR
 
## Also verify from inside Kafka
kubectl exec -n kafka production-cluster-kafka-0 -- \
  bin/kafka-topics.sh \
  --bootstrap-server localhost:9092 \
  --describe \
  --topic orders
## Expected: Shows 6 partitions, each with leader and 3 replicas

Step 4: Build Producer and Consumer Services

Bash

## Deploy a producer service that publishes order events
kubectl apply -f - <<EOF
apiVersion: apps/v1
kind: Deployment
metadata:
  name: order-producer
spec:
  replicas: 2
  selector:
    matchLabels:
      app: order-producer
  template:
    metadata:
      labels:
        app: order-producer
    spec:
      containers:
      * name: producer
        image: confluentinc/cp-kafka:7.6.0
        command:
        * sh
        * -c
        * |
          # Simulate order events being published
          while true; do
            ORDER_ID=$(shuf -i 1000-9999 -n 1)
            AMOUNT=$(shuf -i 100-10000 -n 1)
            USER_ID=$(shuf -i 1-100 -n 1)
            EVENT="{\"order_id\": $ORDER_ID, \"user_id\": $USER_ID, \"amount\": $AMOUNT, \"status\": \"placed\", \"timestamp\": \"$(date -u +%FT%TZ)\"}"
            echo $EVENT | kafka-console-producer \
              --bootstrap-server production-cluster-kafka-bootstrap.kafka:9092 \
              --topic orders \
              --property "key.serializer=org.apache.kafka.common.serialization.StringSerializer"
            echo "Published order $ORDER_ID for user $USER_ID"
            sleep 1
          done
---
## Consumer service that processes orders
apiVersion: apps/v1
kind: Deployment
metadata:
  name: order-consumer
spec:
  replicas: 3  # 3 consumers — each handles 2 of the 6 partitions
  selector:
    matchLabels:
      app: order-consumer
  template:
    metadata:
      labels:
        app: order-consumer
    spec:
      containers:
      * name: consumer
        image: confluentinc/cp-kafka:7.6.0
        command:
        * sh
        * -c
        * |
          kafka-console-consumer \
            --bootstrap-server production-cluster-kafka-bootstrap.kafka:9092 \
            --topic orders \
            --group order-processor \
            --from-beginning
EOF
 
## Verify producer is publishing
kubectl logs deployment/order-producer --tail=10
## Expected: "Published order XXXX for user XX" every second
 
## Verify consumer is processing
kubectl logs deployment/order-consumer --tail=10
## Expected: JSON order events appearing
 
## Check consumer group lag (how far behind is the consumer?)
kubectl exec -n kafka production-cluster-kafka-0 -- \
  bin/kafka-consumer-groups.sh \
  --bootstrap-server localhost:9092 \
  --describe \
  --group order-processor
## Expected: LAG should be small and stable (< 100)
## If LAG is growing, the consumer is falling behind

Step 5: Configure Kafka Connect PostgreSQL Sink

Bash

## Deploy PostgreSQL for the sink
kubectl apply -f - <<EOF
apiVersion: apps/v1
kind: Deployment
metadata:
  name: postgres
spec:
  replicas: 1
  selector:
    matchLabels:
      app: postgres
  template:
    metadata:
      labels:
        app: postgres
    spec:
      containers:
      * name: postgres
        image: postgres:15-alpine
        env:
        * name: POSTGRES_PASSWORD
          value: "kafkapassword"
        * name: POSTGRES_DB
          value: "orders_db"
---
apiVersion: v1
kind: Service
metadata:
  name: postgres
spec:
  selector:
    app: postgres
  ports:
  * port: 5432
EOF
 
## Deploy Kafka Connect with the JDBC connector
kubectl apply -n kafka -f - <<EOF
apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaConnect
metadata:
  name: production-connect
  annotations:
    strimzi.io/use-connector-resources: "true"
spec:
  version: 3.7.0
  replicas: 2
  bootstrapServers: production-cluster-kafka-bootstrap:9092
  config:
    group.id: connect-cluster
    offset.storage.topic: connect-cluster-offsets
    config.storage.topic: connect-cluster-configs
    status.storage.topic: connect-cluster-status
    config.storage.replication.factor: 3
    offset.storage.replication.factor: 3
    status.storage.replication.factor: 3
  build:
    output:
      type: docker
      image: YOUR_ECR/kafka-connect:latest
    plugins:
      * name: jdbc-connector
        artifacts:
          * type: jar
            url: https://packages.confluent.io/maven/io/confluent/kafka-connect-jdbc/10.7.4/kafka-connect-jdbc-10.7.4.jar
EOF
 
## Create the PostgreSQL sink connector
kubectl apply -n kafka -f - <<EOF
apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaConnector
metadata:
  name: orders-postgres-sink
  labels:
    strimzi.io/cluster: production-connect
spec:
  class: io.confluent.connect.jdbc.JdbcSinkConnector
  tasksMax: 3
  config:
    connection.url: jdbc:postgresql://postgres.default:5432/orders_db
    connection.user: postgres
    connection.password: kafkapassword
    topics: orders
    insert.mode: upsert
    pk.mode: record_key
    pk.fields: order_id
    auto.create: true  # Automatically create the orders table
    auto.evolve: true  # Automatically add new columns
EOF
 
## Verify the connector is running
kubectl get kafkaconnector -n kafka
## Expected: orders-postgres-sink showing READY=True, CONNECTOR STATUS=running
 
## Verify data is flowing into PostgreSQL
kubectl exec deployment/postgres -- \
  psql -U postgres -d orders_db \
  -c "SELECT COUNT(*), MAX(timestamp) FROM orders;"
## Expected: Growing row count, timestamp from recent seconds

Step 6: Monitor with Grafana Kafka Dashboard

Bash

## Create Prometheus scrape config for Kafka metrics
kubectl apply -n kafka -f - <<EOF
apiVersion: v1
kind: ConfigMap
metadata:
  name: kafka-metrics
data:
  kafka-metrics-config.yml: |
    # Collect all Kafka JMX metrics
    rules:
    * pattern: kafka.server<type=(.+), name=(.+), clientId=(.+), topic=(.+), partition=(.*)><>Value
      name: kafka_server_$1_$2
      type: GAUGE
      labels:
        clientId: "$3"
        topic: "$4"
        partition: "$5"
    * pattern: kafka.server<type=(.+), name=(.+), clientId=(.+), brokerHost=(.+), brokerPort=(.+)><>Value
      name: kafka_server_$1_$2
      type: GAUGE
      labels:
        clientId: "$3"
        broker: "$4:$5"
EOF
 
## If you have Prometheus and Grafana running (from the monitoring project)
## Import the Strimzi Kafka dashboard
## Dashboard ID: 7589 (Kafka Overview) — import from grafana.com
## Navigate to Grafana -> Dashboards -> Import -> Enter ID 7589
 
## Key metrics to watch in the dashboard:
## - Messages In Per Second (throughput)
## - Consumer Group Lag (are consumers keeping up?)
## - Under Replicated Partitions (should always be 0)
## - Active Controller Count (should always be 1)
## - Offline Partitions Count (should always be 0)

Validation & Testing

Bash

## 1. Verify Kafka cluster health
kubectl get kafka production-cluster -n kafka -o jsonpath='{.status.conditions[0]}'
## Expected: type=Ready, status=True
 
## 2. Check all 3 brokers are in sync
kubectl exec -n kafka production-cluster-kafka-0 -- \
  bin/kafka-metadata-quorum.sh \
  --bootstrap-server localhost:9092 \
  describe --status
## Expected: LeaderId set, all 3 voters showing as healthy
 
## 3. Verify topic replication
kubectl exec -n kafka production-cluster-kafka-0 -- \
  bin/kafka-topics.sh --bootstrap-server localhost:9092 --describe
## Expected: all partitions have 3 replicas, 0 under-replicated
 
## 4. Test Kafka resilience — kill a broker and verify no data loss
kubectl delete pod production-cluster-kafka-1 -n kafka
## Producer should continue without errors
kubectl logs deployment/order-producer --tail=20
## Expected: No errors — Kafka routes to the 2 remaining brokers
 
## 5. Verify broker auto-recovers
kubectl get pods -n kafka --watch
## Expected: production-cluster-kafka-1 restarts and rejoins the cluster
 
## 6. Verify data in PostgreSQL
kubectl exec deployment/postgres -- \
  psql -U postgres -d orders_db \
  -c "SELECT order_id, user_id, amount, status FROM orders ORDER BY timestamp DESC LIMIT 10;"
## Expected: Recent order events with correct data
 
## 7. Check consumer group lag (should be near zero)
kubectl exec -n kafka production-cluster-kafka-0 -- \
  bin/kafka-consumer-groups.sh \
  --bootstrap-server localhost:9092 \
  --describe --all-groups
## Expected: LAG column shows small values close to 0
 
## 8. Measure throughput
kubectl exec -n kafka production-cluster-kafka-0 -- \
  bin/kafka-producer-perf-test.sh \
  --topic orders \
  --num-records 100000 \
  --record-size 1000 \
  --throughput -1 \
  --producer-props bootstrap.servers=localhost:9092
## Expected: 50,000+ records/sec throughput on this setup
echo "Kafka event streaming platform fully operational"

Videos & Guides

Video Guide

Apache Kafka on Kubernetes with Strimzi — Complete Tutorial

Complete Strimzi Kafka operator tutorial covering cluster deployment, topic management, Kafka Connect configuration, consumer groups, and Grafana monitoring dashboard setup.

View Reference

DOCUMENTATION

Strimzi Kafka Operator Official Documentation

Official Strimzi documentation for deploying production Kafka on Kubernetes — cluster configuration, topic operators, Kafka Connect, user management, and security.

View Reference

DOCUMENTATION

Kafka Connect JDBC Connector Documentation

Official Confluent JDBC Sink Connector documentation for automatically writing Kafka topic data to PostgreSQL — upsert modes, auto table creation, and schema evolution.

View Reference