This project deploys a production-grade Apache Kafka event streaming platform on Kubernetes using the Strimzi operator. Kafka is the backbone of event-driven architectures — at Zerodha, every trade placed by every user flows through Kafka before being processed by the matching engine, risk system, and ledger simultaneously. You will build the full event streaming pipeline: a 3-broker Kafka cluster with replication, a producer service that publishes order events, a consumer service that processes them, Kafka Connect to automatically sink processed events into PostgreSQL, and Grafana dashboards to monitor broker health, consumer lag, and throughput. Order Service (producer) | v +-----------+ | Kafka | Broker 0 (kafka-0) | Cluster | Broker 1 (kafka-1) <- 3 brokers, replication factor 3 | (Strimzi) | Broker 2 (kafka-2) +-----------+ | +-----+-----+ | | v v Consumer Kafka Connect Service (sink connector) (processes (writes to PostgreSQL orders) automatically) | | v v Analytics PostgreSQL Dashboard Database
Without an event streaming platform, microservices communicate directly — the order service calls the payment service, the notification service, and the analytics service synchronously. If any downstream service is slow or down, the order service hangs. This is tight coupling and it is fragile at scale. With Kafka, the order service publishes an event and its job is done. The payment service, notification service, and analytics service each consume the event independently. They can be down for hours, catch up when they come back, and the order service never knew anything was wrong. This is how Zerodha processes 10 million orders per day without the trading API slowing down during peak hours.
### Step 1: Install Strimzi Kafka Operator Strimzi is a Kubernetes operator that manages Kafka as a native Kubernetes workload. Instead of manually managing Kafka brokers, you declare what you want in a YAML file and Strimzi handles the rest — rolling upgrades, configuration changes, scaling. ```bash ## Create the Kafka namespace kubectl create namespace kafka ## Install Strimzi operator ## The operator watches for Kafka custom resources and manages the cluster kubectl create -f 'https://strimzi.io/install/latest?namespace=kafka' \ -n kafka ## Wait for the Strimzi operator to be ready kubectl get pods -n kafka --watch ## Expected: strimzi-cluster-operator-xxx Running ## Verify the Kafka CRDs were installed kubectl get crd | grep kafka ## Expected: kafkas.kafka.strimzi.io, kafkatopics.kafka.strimzi.io, ## kafkausers.kafka.strimzi.io, kafkaconnects.kafka.strimzi.io ``` > 📌 **Remember:** The Strimzi operator is the manager. It does not run Kafka itself — it watches for Kafka custom resources and creates the actual Kafka pods. This is the Kubernetes operator pattern — you declare intent, the operator makes it happen. ### Step 2: Deploy a Production 3-Broker Kafka Cluster ```bash ## Create the Kafka cluster custom resource kubectl apply -n kafka -f - <<EOF apiVersion: kafka.strimzi.io/v1beta2 kind: Kafka metadata: name: production-cluster spec: kafka: version: 3.7.0 replicas: 3 # 3 brokers for high availability listeners: * name: plain port: 9092 type: internal tls: false * name: tls port: 9093 type: internal tls: true config: # Replication and durability settings offsets.topic.replication.factor: 3 transaction.state.log.replication.factor: 3 transaction.state.log.min.isr: 2 # Minimum in-sync replicas default.replication.factor: 3 min.insync.replicas: 2 # Performance tuning num.network.threads: 3 num.io.threads: 8 socket.send.buffer.bytes: 102400 socket.receive.buffer.bytes: 102400 socket.request.max.bytes: 104857600 # Log retention log.retention.hours: 168 # Keep messages for 7 days log.retention.bytes: 107374182400 # 100GB per partition max log.segment.bytes: 1073741824 # 1GB segments storage: type: jbod volumes: * id: 0 type: persistent-claim size: 100Gi deleteClaim: false resources: requests: memory: 2Gi cpu: 500m limits: memory: 4Gi cpu: 2 metricsConfig: type: jmxPrometheusExporter valueFrom: configMapKeyRef: name: kafka-metrics key: kafka-metrics-config.yml zookeeper: replicas: 3 # Zookeeper ensemble for cluster coordination storage: type: persistent-claim size: 10Gi deleteClaim: false resources: requests: memory: 1Gi cpu: 250m entityOperator: topicOperator: {} # Manages KafkaTopic resources userOperator: {} # Manages KafkaUser resources EOF ## This takes 3-5 minutes for all brokers to start kubectl get kafka -n kafka --watch ## Expected: production-cluster shows READY=True ## Verify all 3 brokers are running kubectl get pods -n kafka -l strimzi.io/name=production-cluster-kafka ## Expected: production-cluster-kafka-0, -1, -2 all Running 1/1 ``` ### Step 3: Create Topics with Proper Configuration ```bash ## Create the orders topic using Strimzi's KafkaTopic custom resource ## This is better than using kafka-topics.sh because: ## 1. It is declarative and version-controlled ## 2. Strimzi reconciles the topic if it drifts ## 3. Config changes can be applied via kubectl kubectl apply -n kafka -f - <<EOF apiVersion: kafka.strimzi.io/v1beta2 kind: KafkaTopic metadata: name: orders labels: strimzi.io/cluster: production-cluster # Must match the Kafka cluster name spec: partitions: 6 # 6 partitions for parallelism (2 per broker) replicas: 3 # Each partition replicated on all 3 brokers config: retention.ms: 604800000 # 7 days in milliseconds cleanup.policy: delete # Delete old messages (not compact) min.insync.replicas: "2" # Producer must get ack from at least 2 replicas compression.type: lz4 # Compress messages for storage efficiency --- apiVersion: kafka.strimzi.io/v1beta2 kind: KafkaTopic metadata: name: payments labels: strimzi.io/cluster: production-cluster spec: partitions: 6 replicas: 3 config: retention.ms: 2592000000 # 30 days — payment events kept longer min.insync.replicas: "2" --- apiVersion: kafka.strimzi.io/v1beta2 kind: KafkaTopic metadata: name: notifications labels: strimzi.io/cluster: production-cluster spec: partitions: 3 # Lower volume — fewer partitions replicas: 3 config: retention.ms: 86400000 # 1 day — notification events expire quickly EOF ## Verify topics were created kubectl get kafkatopics -n kafka ## Expected: orders, payments, notifications all showing READY=True, PARTITIONS, REPLICATION FACTOR ## Also verify from inside Kafka kubectl exec -n kafka production-cluster-kafka-0 -- \ bin/kafka-topics.sh \ --bootstrap-server localhost:9092 \ --describe \ --topic orders ## Expected: Shows 6 partitions, each with leader and 3 replicas ``` ### Step 4: Build Producer and Consumer Services ```bash ## Deploy a producer service that publishes order events kubectl apply -f - <<EOF apiVersion: apps/v1 kind: Deployment metadata: name: order-producer spec: replicas: 2 selector: matchLabels: app: order-producer template: metadata: labels: app: order-producer spec: containers: * name: producer image: confluentinc/cp-kafka:7.6.0 command: * sh * -c * | # Simulate order events being published while true; do ORDER_ID=$(shuf -i 1000-9999 -n 1) AMOUNT=$(shuf -i 100-10000 -n 1) USER_ID=$(shuf -i 1-100 -n 1) EVENT="{\"order_id\": $ORDER_ID, \"user_id\": $USER_ID, \"amount\": $AMOUNT, \"status\": \"placed\", \"timestamp\": \"$(date -u +%FT%TZ)\"}" echo $EVENT | kafka-console-producer \ --bootstrap-server production-cluster-kafka-bootstrap.kafka:9092 \ --topic orders \ --property "key.serializer=org.apache.kafka.common.serialization.StringSerializer" echo "Published order $ORDER_ID for user $USER_ID" sleep 1 done --- ## Consumer service that processes orders apiVersion: apps/v1 kind: Deployment metadata: name: order-consumer spec: replicas: 3 # 3 consumers — each handles 2 of the 6 partitions selector: matchLabels: app: order-consumer template: metadata: labels: app: order-consumer spec: containers: * name: consumer image: confluentinc/cp-kafka:7.6.0 command: * sh * -c * | kafka-console-consumer \ --bootstrap-server production-cluster-kafka-bootstrap.kafka:9092 \ --topic orders \ --group order-processor \ --from-beginning EOF ## Verify producer is publishing kubectl logs deployment/order-producer --tail=10 ## Expected: "Published order XXXX for user XX" every second ## Verify consumer is processing kubectl logs deployment/order-consumer --tail=10 ## Expected: JSON order events appearing ## Check consumer group lag (how far behind is the consumer?) kubectl exec -n kafka production-cluster-kafka-0 -- \ bin/kafka-consumer-groups.sh \ --bootstrap-server localhost:9092 \ --describe \ --group order-processor ## Expected: LAG should be small and stable (< 100) ## If LAG is growing, the consumer is falling behind ``` ### Step 5: Configure Kafka Connect PostgreSQL Sink ```bash ## Deploy PostgreSQL for the sink kubectl apply -f - <<EOF apiVersion: apps/v1 kind: Deployment metadata: name: postgres spec: replicas: 1 selector: matchLabels: app: postgres template: metadata: labels: app: postgres spec: containers: * name: postgres image: postgres:15-alpine env: * name: POSTGRES_PASSWORD value: "kafkapassword" * name: POSTGRES_DB value: "orders_db" --- apiVersion: v1 kind: Service metadata: name: postgres spec: selector: app: postgres ports: * port: 5432 EOF ## Deploy Kafka Connect with the JDBC connector kubectl apply -n kafka -f - <<EOF apiVersion: kafka.strimzi.io/v1beta2 kind: KafkaConnect metadata: name: production-connect annotations: strimzi.io/use-connector-resources: "true" spec: version: 3.7.0 replicas: 2 bootstrapServers: production-cluster-kafka-bootstrap:9092 config: group.id: connect-cluster offset.storage.topic: connect-cluster-offsets config.storage.topic: connect-cluster-configs status.storage.topic: connect-cluster-status config.storage.replication.factor: 3 offset.storage.replication.factor: 3 status.storage.replication.factor: 3 build: output: type: docker image: YOUR_ECR/kafka-connect:latest plugins: * name: jdbc-connector artifacts: * type: jar url: https://packages.confluent.io/maven/io/confluent/kafka-connect-jdbc/10.7.4/kafka-connect-jdbc-10.7.4.jar EOF ## Create the PostgreSQL sink connector kubectl apply -n kafka -f - <<EOF apiVersion: kafka.strimzi.io/v1beta2 kind: KafkaConnector metadata: name: orders-postgres-sink labels: strimzi.io/cluster: production-connect spec: class: io.confluent.connect.jdbc.JdbcSinkConnector tasksMax: 3 config: connection.url: jdbc:postgresql://postgres.default:5432/orders_db connection.user: postgres connection.password: kafkapassword topics: orders insert.mode: upsert pk.mode: record_key pk.fields: order_id auto.create: true # Automatically create the orders table auto.evolve: true # Automatically add new columns EOF ## Verify the connector is running kubectl get kafkaconnector -n kafka ## Expected: orders-postgres-sink showing READY=True, CONNECTOR STATUS=running ## Verify data is flowing into PostgreSQL kubectl exec deployment/postgres -- \ psql -U postgres -d orders_db \ -c "SELECT COUNT(*), MAX(timestamp) FROM orders;" ## Expected: Growing row count, timestamp from recent seconds ``` ### Step 6: Monitor with Grafana Kafka Dashboard ```bash ## Create Prometheus scrape config for Kafka metrics kubectl apply -n kafka -f - <<EOF apiVersion: v1 kind: ConfigMap metadata: name: kafka-metrics data: kafka-metrics-config.yml: | # Collect all Kafka JMX metrics rules: * pattern: kafka.server<type=(.+), name=(.+), clientId=(.+), topic=(.+), partition=(.*)><>Value name: kafka_server_$1_$2 type: GAUGE labels: clientId: "$3" topic: "$4" partition: "$5" * pattern: kafka.server<type=(.+), name=(.+), clientId=(.+), brokerHost=(.+), brokerPort=(.+)><>Value name: kafka_server_$1_$2 type: GAUGE labels: clientId: "$3" broker: "$4:$5" EOF ## If you have Prometheus and Grafana running (from the monitoring project) ## Import the Strimzi Kafka dashboard ## Dashboard ID: 7589 (Kafka Overview) — import from grafana.com ## Navigate to Grafana -> Dashboards -> Import -> Enter ID 7589 ## Key metrics to watch in the dashboard: ## - Messages In Per Second (throughput) ## - Consumer Group Lag (are consumers keeping up?) ## - Under Replicated Partitions (should always be 0) ## - Active Controller Count (should always be 1) ## - Offline Partitions Count (should always be 0) ```
```bash ## 1. Verify Kafka cluster health kubectl get kafka production-cluster -n kafka -o jsonpath='{.status.conditions[0]}' ## Expected: type=Ready, status=True ## 2. Check all 3 brokers are in sync kubectl exec -n kafka production-cluster-kafka-0 -- \ bin/kafka-metadata-quorum.sh \ --bootstrap-server localhost:9092 \ describe --status ## Expected: LeaderId set, all 3 voters showing as healthy ## 3. Verify topic replication kubectl exec -n kafka production-cluster-kafka-0 -- \ bin/kafka-topics.sh --bootstrap-server localhost:9092 --describe ## Expected: all partitions have 3 replicas, 0 under-replicated ## 4. Test Kafka resilience — kill a broker and verify no data loss kubectl delete pod production-cluster-kafka-1 -n kafka ## Producer should continue without errors kubectl logs deployment/order-producer --tail=20 ## Expected: No errors — Kafka routes to the 2 remaining brokers ## 5. Verify broker auto-recovers kubectl get pods -n kafka --watch ## Expected: production-cluster-kafka-1 restarts and rejoins the cluster ## 6. Verify data in PostgreSQL kubectl exec deployment/postgres -- \ psql -U postgres -d orders_db \ -c "SELECT order_id, user_id, amount, status FROM orders ORDER BY timestamp DESC LIMIT 10;" ## Expected: Recent order events with correct data ## 7. Check consumer group lag (should be near zero) kubectl exec -n kafka production-cluster-kafka-0 -- \ bin/kafka-consumer-groups.sh \ --bootstrap-server localhost:9092 \ --describe --all-groups ## Expected: LAG column shows small values close to 0 ## 8. Measure throughput kubectl exec -n kafka production-cluster-kafka-0 -- \ bin/kafka-producer-perf-test.sh \ --topic orders \ --num-records 100000 \ --record-size 1000 \ --throughput -1 \ --producer-props bootstrap.servers=localhost:9092 ## Expected: 50,000+ records/sec throughput on this setup echo "Kafka event streaming platform fully operational" ```
This project deploys a production-grade Apache Kafka event streaming platform on Kubernetes using the Strimzi operator. ...
Without an event streaming platform, microservices communicate directly — the order service calls the payment service, t...
Step 1: Install Strimzi Kafka Operator Strimzi is a Kubernetes operator that manages Kafka as a native Kubernetes worklo...
...
Aligns directly with DevOps, Site Reliability (SRE), and Platform Engineering job descriptions.