Deploy Kafka on Kubernetes with Strimzi

Deploy a 3-broker KRaft Kafka cluster with Strimzi, build keyed producer and consumer apps, sink to Postgres with Kafka Connect, monitor lag in Grafana.

Architecture Overview & Problem Statement

This project deploys a replicated Apache Kafka cluster on Kubernetes with the Strimzi operator in KRaft mode (no ZooKeeper), then builds the pipeline around it: a producer publishes order events, a consumer group processes them, Kafka Connect sinks them into PostgreSQL, and Prometheus and Grafana show broker health and consumer lag.

What you build, learn and can show

Build: a 3-broker KRaft cluster, replicated topics, a keyed idempotent producer, a consumer group, a JDBC sink connector into PostgreSQL and a monitoring stack
Learn: how Kubernetes operators manage Kafka declaratively, how replication factor, min.insync.replicas and acks=all work together to prevent data loss, and how to read consumer lag
Show: a repository with the manifests, a broker-failure test that loses no data, and a Grafana view of lag and under-replicated partitions

flowchart LR P[Order producer
2 pods] --> K[Kafka cluster
3 brokers] K --> C[Order consumer
3 pods] K --> KC[Kafka Connect
JDBC sink] KC --> PG[(PostgreSQL)] K -.->|JMX metrics| PR[Prometheus] PR --> G[Grafana] classDef blue fill:#326CE5,color:#ffffff classDef orange fill:#F97316,color:#ffffff classDef green fill:#10B981,color:#ffffff classDef purple fill:#8B5CF6,color:#ffffff class P,C blue class K orange class KC,PG green class PR,G purple

The problem this solves

Without an event streaming platform, microservices call each other directly. The order service calls payments, notifications and analytics synchronously, so if any one is slow or down, the order service hangs. That is tight coupling, and it gets more fragile as you add services.

With Kafka the order service publishes an event and is done. A brokerage like Zerodha or a food-delivery app like Swiggy has the same shape: one order event, many independent consumers such as risk, ledger, notifications and analytics. Each consumer reads at its own pace and can be down for hours and catch up later, without the order service noticing.

Which versions this was written against

Strimzi 0.46 and later removed ZooKeeper support entirely, so a modern Strimzi deployment is KRaft only and uses KafkaNodePool resources. This project was written against Strimzi 0.47.0 and Apache Kafka 4.0.0. If a newer Strimzi exists when you build it, check its release notes for supported Kafka versions and update the two version values below together.

Prerequisites

A cluster with enough room for three Kafka nodes, Connect, PostgreSQL and a monitoring stack. Plan for roughly 12 GB of free memory, or reduce replicas for a smaller lab (see Milestone 2).
A container registry you can push to (Docker Hub, GHCR or ECR). Kafka Connect builds a custom image and needs somewhere to store it.
kubectl, helm, curl and openssl

Milestone 1: Install the Strimzi Operator

This milestone installs Strimzi, the operator that turns Kafka custom resources into running brokers.

Strimzi manages Kafka as a native Kubernetes workload. You declare what you want in YAML and the operator handles rolling upgrades, configuration changes and scaling. The operator itself does not run Kafka. It watches for Kafka resources and creates the pods.

Bash

export STRIMZI_VERSION=0.47.0
 
kubectl create namespace kafka
 
curl -sL "https://github.com/strimzi/strimzi-kafka-operator/releases/download/${STRIMZI_VERSION}/strimzi-cluster-operator-${STRIMZI_VERSION}.yaml" \
    | sed 's/namespace: .*/namespace: kafka/' \
    | kubectl create -f - -n kafka
 
kubectl rollout status deployment/strimzi-cluster-operator -n kafka
 
kubectl get crd | grep strimzi
## Expect: kafkas, kafkanodepools, kafkatopics, kafkausers,
##         kafkaconnects and kafkaconnectors under strimzi.io

Remember
This is the Kubernetes operator pattern: you declare intent, the operator makes it true and keeps it true.

Milestone 2: Deploy a 3-Broker KRaft Kafka Cluster

This milestone deploys three Kafka nodes that act as both controllers and brokers, with replication settings that survive the loss of one node.

In KRaft mode Kafka stores its own metadata, so there is no separate ZooKeeper ensemble. A KafkaNodePool defines the nodes and their roles, and the Kafka resource defines the cluster. Three combined controller and broker nodes is a good lab and small-production shape. Larger production clusters usually split controllers from brokers.

First the metrics configuration, so Prometheus can read broker health later:

Bash

kubectl apply -n kafka -f - <<'EOF'
apiVersion: v1
kind: ConfigMap
metadata:
  name: kafka-metrics
data:
  kafka-metrics-config.yml: |
    lowercaseOutputName: true
    rules:
      - pattern: kafka.server<type=(.+), name=(.+)PerSec\w*><>Count
        name: kafka_server_$1_$2_total
      - pattern: kafka.server<type=BrokerTopicMetrics, name=(.+)PerSec, topic=(.+)><>Count
        name: kafka_server_brokertopicmetrics_$1_total
        labels:
          topic: "$2"
      - pattern: kafka.server<type=ReplicaManager, name=(.+)><>Value
        name: kafka_server_replicamanager_$1
      - pattern: kafka.controller<type=KafkaController, name=(.+)><>Value
        name: kafka_controller_kafkacontroller_$1
EOF

Now the node pool and cluster:

Bash

kubectl apply -n kafka -f - <<'EOF'
apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaNodePool
metadata:
  name: dual-role
  labels:
    strimzi.io/cluster: production-cluster
spec:
  replicas: 3
  roles:
    - controller
    - broker
  storage:
    type: persistent-claim
    size: 20Gi
    deleteClaim: false
  resources:
    requests:
      memory: 1Gi
      cpu: 250m
    limits:
      memory: 2Gi
      cpu: "1"
---
apiVersion: kafka.strimzi.io/v1beta2
kind: Kafka
metadata:
  name: production-cluster
  annotations:
    strimzi.io/node-pools: enabled
    strimzi.io/kraft: enabled
spec:
  kafka:
    version: 4.0.0
    metadataVersion: 4.0-IV3
    listeners:
      - name: plain
        port: 9092
        type: internal
        tls: false
      - name: tls
        port: 9093
        type: internal
        tls: true
    config:
      offsets.topic.replication.factor: 3
      transaction.state.log.replication.factor: 3
      transaction.state.log.min.isr: 2
      default.replication.factor: 3
      min.insync.replicas: 2
      auto.create.topics.enable: false
    metricsConfig:
      type: jmxPrometheusExporter
      valueFrom:
        configMapKeyRef:
          name: kafka-metrics
          key: kafka-metrics-config.yml
  entityOperator:
    topicOperator: {}
    userOperator: {}
  kafkaExporter:
    topicRegex: ".*"
    groupRegex: ".*"
EOF
 
kubectl wait kafka/production-cluster --for=condition=Ready --timeout=600s -n kafka
kubectl get pods -n kafka -l strimzi.io/cluster=production-cluster
## Expect production-cluster-dual-role-0, -1, -2 and the entity operator and exporter pods Running

The three settings that protect your data: default.replication.factor: 3 keeps three copies, min.insync.replicas: 2 refuses writes when fewer than two copies are in sync, and the producer later uses acks=all. Together they mean an acknowledged write exists on at least two brokers.

Security
The plain listener has no encryption or authentication and is acceptable only in a lab. For anything real, use the TLS listener with KafkaUser resources and authorization. Strimzi's user operator manages those declaratively.

Tip
For a small lab, set replicas: 3 but lower the memory requests, or use a single-node pool and replication.factor: 1 while you learn. You will not be able to run the failure test in the Validation step without three nodes.

Milestone 3: Create Topics with Proper Configuration

This milestone declares three topics as Kubernetes resources, with partitions and retention chosen deliberately.

Creating topics with a KafkaTopic resource beats kafka-topics.sh because the topic is declarative and version-controlled, and Strimzi reconciles it if it drifts. Auto-creation is turned off in the cluster config, so a typo in a client topic name fails loudly instead of silently creating a new topic.

Bash

kubectl apply -n kafka -f - <<'EOF'
apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaTopic
metadata:
  name: orders
  labels:
    strimzi.io/cluster: production-cluster
spec:
  partitions: 6
  replicas: 3
  config:
    retention.ms: 604800000
    cleanup.policy: delete
    min.insync.replicas: "2"
    compression.type: lz4
---
apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaTopic
metadata:
  name: payments
  labels:
    strimzi.io/cluster: production-cluster
spec:
  partitions: 6
  replicas: 3
  config:
    retention.ms: 2592000000
    min.insync.replicas: "2"
---
apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaTopic
metadata:
  name: notifications
  labels:
    strimzi.io/cluster: production-cluster
spec:
  partitions: 3
  replicas: 3
  config:
    retention.ms: 86400000
EOF
 
kubectl get kafkatopics -n kafka
 
kubectl exec -n kafka production-cluster-dual-role-0 -- \
    bin/kafka-topics.sh --bootstrap-server localhost:9092 --describe --topic orders
## Expect 6 partitions, each with a leader, 3 replicas and 3 in-sync replicas (Isr)

Why these numbers: six partitions let up to six consumers in one group work in parallel and spread leaders across three brokers. Payment events keep 30 days because they are audited later. Notifications keep 1 day because stale notifications are worthless.

Tip
Messages with the same key always go to the same partition, which preserves per-key ordering. Choose the key (here the order id) based on what must stay in order.

Milestone 4: Build Producer and Consumer Services

This milestone deploys a keyed, idempotent producer and a consumer group, both written as small Python programs, and checks the lag.

The producer uses acks=all and idempotence, so a retry after a broker failure cannot create duplicates. It sends each event with a schema envelope, because the JDBC sink in the next milestone needs to know column types. The consumer commits offsets only after it has processed a message, which gives at-least-once delivery.

Bash

mkdir -p apps
 
cat > apps/producer.py << 'EOF'
import json
import os
import random
import time
from datetime import datetime, timezone
 
from confluent_kafka import Producer
 
producer = Producer({
    "bootstrap.servers": os.environ["BOOTSTRAP"],
    "acks": "all",
    "enable.idempotence": True,
    "compression.type": "lz4",
    "linger.ms": 20,
})
 
SCHEMA = {
    "type": "struct",
    "name": "order",
    "optional": False,
    "fields": [
        {"field": "order_id", "type": "int64", "optional": False},
        {"field": "user_id", "type": "int32", "optional": False},
        {"field": "amount", "type": "int32", "optional": False},
        {"field": "status", "type": "string", "optional": False},
        {"field": "event_time", "type": "string", "optional": False},
    ],
}
 
 
def on_delivery(err, msg):
    if err is not None:
        print(f"DELIVERY FAILED: {err}", flush=True)
 
 
while True:
    order_id = time.time_ns() // 1000
    event = {
        "order_id": order_id,
        "user_id": random.randint(1, 100),
        "amount": random.randint(100, 10000),
        "status": "placed",
        "event_time": datetime.now(timezone.utc).isoformat(),
    }
    producer.produce(
        "orders",
        key=str(order_id),
        value=json.dumps({"schema": SCHEMA, "payload": event}),
        on_delivery=on_delivery,
    )
    producer.poll(0)
    print(f"produced order {order_id}", flush=True)
    time.sleep(1)
EOF
 
cat > apps/consumer.py << 'EOF'
import json
import os
 
from confluent_kafka import Consumer
 
consumer = Consumer({
    "bootstrap.servers": os.environ["BOOTSTRAP"],
    "group.id": "order-processor",
    "auto.offset.reset": "earliest",
    "enable.auto.commit": False,
})
consumer.subscribe(["orders"])
 
while True:
    msg = consumer.poll(1.0)
    if msg is None:
        continue
    if msg.error():
        print(f"consumer error: {msg.error()}", flush=True)
        continue
    order = json.loads(msg.value())["payload"]
    print(f"partition={msg.partition()} offset={msg.offset()} "
          f"order={order['order_id']} amount={order['amount']}", flush=True)
    # Commit only after processing: at-least-once delivery.
    # In real services, commit in batches to cut overhead.
    consumer.commit(msg=msg, asynchronous=False)
EOF
 
kubectl create namespace orders-demo
kubectl create configmap orders-apps -n orders-demo \
    --from-file=apps/producer.py --from-file=apps/consumer.py

Bash

kubectl apply -n orders-demo -f - <<'EOF'
apiVersion: apps/v1
kind: Deployment
metadata:
  name: order-producer
spec:
  replicas: 2
  selector:
    matchLabels:
      app: order-producer
  template:
    metadata:
      labels:
        app: order-producer
    spec:
      containers:
        - name: producer
          image: python:3.12-slim
          command:
            - sh
            - -c
            - "pip install --no-cache-dir confluent-kafka==2.5.0 && python /app/producer.py"
          env:
            - name: BOOTSTRAP
              value: production-cluster-kafka-bootstrap.kafka.svc:9092
          volumeMounts:
            - name: app
              mountPath: /app
      volumes:
        - name: app
          configMap:
            name: orders-apps
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: order-consumer
spec:
  replicas: 3
  selector:
    matchLabels:
      app: order-consumer
  template:
    metadata:
      labels:
        app: order-consumer
    spec:
      containers:
        - name: consumer
          image: python:3.12-slim
          command:
            - sh
            - -c
            - "pip install --no-cache-dir confluent-kafka==2.5.0 && python /app/consumer.py"
          env:
            - name: BOOTSTRAP
              value: production-cluster-kafka-bootstrap.kafka.svc:9092
          volumeMounts:
            - name: app
              mountPath: /app
      volumes:
        - name: app
          configMap:
            name: orders-apps
EOF
 
kubectl logs -n orders-demo deployment/order-producer --tail=5
## Expect "produced order ..." every second
 
kubectl logs -n orders-demo deployment/order-consumer --tail=5
## Expect partition, offset and order lines
 
kubectl exec -n kafka production-cluster-dual-role-0 -- \
    bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 \
    --describe --group order-processor
## LAG should be small and stable. Growing LAG means the consumers are falling behind.

The three consumer replicas share the six partitions, two each. A fourth to seventh consumer would sit idle, because a partition is read by at most one consumer in a group.

Common Mistake
Running more consumers than partitions and expecting more throughput. Extra consumers in a group stay idle. Scale partitions first, and remember partition count is hard to reduce later.

Tip
The pip install at container start keeps this lab short. In a real service you bake dependencies into an image so a PyPI outage cannot stop a pod from starting.

Milestone 5: Sink Orders into PostgreSQL with Kafka Connect

This milestone deploys PostgreSQL and a Kafka Connect cluster with the JDBC connector, so every order event lands in a database table automatically.

Kafka Connect moves data between Kafka and other systems without custom code. The JDBC sink connector reads the schema envelope the producer sends and creates the table for you. Passwords stay out of the manifests: Connect mounts a Kubernetes Secret and reads it through a config provider.

Deploy PostgreSQL

Bash

export PG_PASSWORD="$(openssl rand -hex 16)"
 
kubectl create secret generic postgres-credentials -n orders-demo \
    --from-literal=password="$PG_PASSWORD"
 
kubectl create secret generic pg-connect-credentials -n kafka \
    --from-literal=username=postgres \
    --from-literal=password="$PG_PASSWORD"
 
kubectl apply -n orders-demo -f - <<'EOF'
apiVersion: apps/v1
kind: Deployment
metadata:
  name: postgres
spec:
  replicas: 1
  selector:
    matchLabels:
      app: postgres
  template:
    metadata:
      labels:
        app: postgres
    spec:
      containers:
        - name: postgres
          image: postgres:16-alpine
          env:
            - name: POSTGRES_DB
              value: orders_db
            - name: POSTGRES_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: postgres-credentials
                  key: password
          ports:
            - containerPort: 5432
---
apiVersion: v1
kind: Service
metadata:
  name: postgres
spec:
  selector:
    app: postgres
  ports:
    - port: 5432
EOF

This Postgres has no persistent volume, so data disappears if the pod restarts. That is fine for a lab and wrong for anything else.

Deploy Kafka Connect with the JDBC plugin

Strimzi builds a custom Connect image containing the plugin and pushes it to your registry. Replace the three registry placeholders with your own.

Bash

kubectl create secret docker-registry registry-credentials -n kafka \
    --docker-server="$REGISTRY_SERVER" \
    --docker-username="$REGISTRY_USER" \
    --docker-password="$REGISTRY_TOKEN"
 
kubectl apply -n kafka -f - <<EOF
apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaConnect
metadata:
  name: production-connect
  annotations:
    strimzi.io/use-connector-resources: "true"
spec:
  version: 4.0.0
  replicas: 2
  bootstrapServers: production-cluster-kafka-bootstrap:9092
  config:
    group.id: connect-cluster
    offset.storage.topic: connect-cluster-offsets
    config.storage.topic: connect-cluster-configs
    status.storage.topic: connect-cluster-status
    offset.storage.replication.factor: 3
    config.storage.replication.factor: 3
    status.storage.replication.factor: 3
    key.converter: org.apache.kafka.connect.storage.StringConverter
    value.converter: org.apache.kafka.connect.json.JsonConverter
    value.converter.schemas.enable: true
    config.providers: directory
    config.providers.directory.class: org.apache.kafka.common.config.provider.DirectoryConfigProvider
  build:
    output:
      type: docker
      image: ${REGISTRY_SERVER}/${REGISTRY_USER}/kafka-connect-jdbc:1.0.0
      pushSecret: registry-credentials
    plugins:
      - name: jdbc-connector
        artifacts:
          - type: zip
            url: https://hub-downloads.confluent.io/api/plugins/confluentinc/kafka-connect-jdbc/versions/10.7.4/confluentinc-kafka-connect-jdbc-10.7.4.zip
  template:
    pod:
      volumes:
        - name: pg-credentials
          secret:
            secretName: pg-connect-credentials
    connectContainer:
      volumeMounts:
        - name: pg-credentials
          mountPath: /opt/kafka/external-configuration/pg-credentials
EOF
 
kubectl wait kafkaconnect/production-connect --for=condition=Ready --timeout=600s -n kafka

The Confluent JDBC connector is free to use under the Confluent Community License, which restricts offering it as a competing hosted service. If you need a fully Apache-licensed option, the Aiven JDBC connector is a common alternative.

Create the sink connector

Bash

kubectl apply -n kafka -f - <<'EOF'
apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaConnector
metadata:
  name: orders-postgres-sink
  labels:
    strimzi.io/cluster: production-connect
spec:
  class: io.confluent.connect.jdbc.JdbcSinkConnector
  tasksMax: 3
  config:
    topics: orders
    connection.url: jdbc:postgresql://postgres.orders-demo.svc:5432/orders_db
    connection.user: ${directory:/opt/kafka/external-configuration/pg-credentials:username}
    connection.password: ${directory:/opt/kafka/external-configuration/pg-credentials:password}
    table.name.format: orders
    insert.mode: upsert
    pk.mode: record_value
    pk.fields: order_id
    auto.create: true
    auto.evolve: true
EOF
 
kubectl get kafkaconnector -n kafka
## Expect orders-postgres-sink READY True
 
kubectl exec -n orders-demo deployment/postgres -- \
    psql -U postgres -d orders_db \
    -c "SELECT COUNT(*), MAX(event_time) FROM orders;"
## Expect a growing count and a recent timestamp

upsert with pk.fields: order_id makes the sink idempotent: if Kafka redelivers an event, the row is overwritten rather than duplicated.

Common Mistake
Sending plain JSON with no schema and pointing the JDBC sink at it. The sink needs a schema to create columns and map types, and fails with an error about missing schemas. Either use the schema envelope as here, or use a schema registry with Avro.

Security
Never put a database password in a connector config or manifest. Mount a Secret and reference it with the directory provider, as above.

Milestone 6: Monitor Kafka with Prometheus and Grafana

This milestone installs Prometheus and Grafana, scrapes the brokers and the exporter, and shows the metrics that matter in a dashboard.

Bash

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo update
 
kubectl create namespace monitoring
export GRAFANA_PASSWORD="$(openssl rand -base64 18)"
echo "Grafana admin password: $GRAFANA_PASSWORD"
 
helm install kube-prometheus-stack \
    prometheus-community/kube-prometheus-stack \
    --namespace monitoring \
    --set prometheus.prometheusSpec.podMonitorSelectorNilUsesHelmValues=false \
    --set prometheus.prometheusSpec.serviceMonitorSelectorNilUsesHelmValues=false \
    --set grafana.adminPassword="$GRAFANA_PASSWORD" \
    --wait

A single PodMonitor covers both the brokers and the Kafka exporter, which share the Strimzi labels and a metrics port named tcp-prometheus.

Bash

kubectl apply -n kafka -f - <<'EOF'
apiVersion: monitoring.coreos.com/v1
kind: PodMonitor
metadata:
  name: strimzi-kafka
spec:
  selector:
    matchExpressions:
      - key: strimzi.io/kind
        operator: In
        values: [Kafka]
  podMetricsEndpoints:
    - path: /metrics
      port: tcp-prometheus
EOF

Import the Strimzi dashboards from the project repository at the same version you installed. Grafana's sidecar loads any ConfigMap labelled grafana_dashboard=1.

Bash

BASE="https://raw.githubusercontent.com/strimzi/strimzi-kafka-operator/${STRIMZI_VERSION}/examples/metrics/grafana-dashboards"
 
curl -sL "$BASE/strimzi-kafka.json" -o strimzi-kafka.json
curl -sL "$BASE/strimzi-kafka-exporter.json" -o strimzi-kafka-exporter.json
 
kubectl create configmap strimzi-kafka-dashboards -n monitoring \
    --from-file=strimzi-kafka.json --from-file=strimzi-kafka-exporter.json
kubectl label configmap strimzi-kafka-dashboards -n monitoring grafana_dashboard=1
 
kubectl port-forward svc/kube-prometheus-stack-grafana 3000:80 -n monitoring

Open http://localhost:3000 (user admin). If panels say the data source is missing, select the Prometheus data source from the dropdown at the top of the dashboard.

The metrics to watch, and the PromQL to run if a panel is empty:

What	Healthy value	PromQL
Under-replicated partitions	0	`sum(kafka_server_replicamanager_underreplicatedpartitions)`
Active controller count	1	`sum(kafka_controller_kafkacontroller_activecontrollercount)`
Offline partitions	0	`sum(kafka_controller_kafkacontroller_offlinepartitionscount)`
Consumer lag	small and stable	`sum by (topic) (kafka_consumergroup_lag{consumergroup="order-processor"})`
Messages in per second	tracks producer rate	`sum(rate(kafka_server_brokertopicmetrics_messagesin_total[1m]))`

Tip
Alert on sustained consumer lag growth, not on a single high reading. Lag naturally spikes during rebalances and recovers on its own.

Validation & Testing

This final step proves the cluster is healthy, the data flows end to end and a broker failure loses nothing.

Bash

## 1. Cluster is Ready
kubectl get kafka production-cluster -n kafka \
    -o jsonpath='{.status.conditions[?(@.type=="Ready")].status}'
## Expect: True
 
## 2. KRaft quorum is healthy
kubectl exec -n kafka production-cluster-dual-role-0 -- \
    bin/kafka-metadata-quorum.sh --bootstrap-server localhost:9092 describe --status
## Expect a LeaderId and 3 voters
 
## 3. Topics are fully replicated
kubectl exec -n kafka production-cluster-dual-role-0 -- \
    bin/kafka-topics.sh --bootstrap-server localhost:9092 --describe --under-replicated-partitions
## Expect no output
 
## 4. Data is reaching PostgreSQL
kubectl exec -n orders-demo deployment/postgres -- \
    psql -U postgres -d orders_db \
    -c "SELECT order_id, user_id, amount, status FROM orders ORDER BY event_time DESC LIMIT 5;"
 
## 5. Consumer lag is small
kubectl exec -n kafka production-cluster-dual-role-0 -- \
    bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --all-groups

Broker failure test

Kill one broker while the producer runs, and check that nothing was lost. With three replicas, min.insync.replicas=2 and acks=all, the cluster keeps accepting writes with one broker down.

Bash

## Count rows before
kubectl exec -n orders-demo deployment/postgres -- \
    psql -U postgres -d orders_db -tc "SELECT COUNT(*) FROM orders;"
 
kubectl delete pod production-cluster-dual-role-1 -n kafka
 
## Producers should log no DELIVERY FAILED lines
kubectl logs -n orders-demo deployment/order-producer --tail=30 | grep -c "DELIVERY FAILED"
## Expect: 0
 
## Watch the broker rejoin, then confirm replication recovers
kubectl get pods -n kafka -w
kubectl exec -n kafka production-cluster-dual-role-0 -- \
    bin/kafka-topics.sh --bootstrap-server localhost:9092 --describe --under-replicated-partitions
## Expect no output once the broker is back in sync
 
## Count rows after: it should keep growing with no gaps in your producer logs

Measure throughput on a separate topic

Run the performance test on its own topic. If you ran it against orders, the sink would try to insert test records and fail.

Bash

kubectl apply -n kafka -f - <<'EOF'
apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaTopic
metadata:
  name: perf-test
  labels:
    strimzi.io/cluster: production-cluster
spec:
  partitions: 6
  replicas: 3
EOF
 
kubectl exec -n kafka production-cluster-dual-role-0 -- \
    bin/kafka-producer-perf-test.sh \
    --topic perf-test \
    --num-records 100000 \
    --record-size 1000 \
    --throughput -1 \
    --producer-props bootstrap.servers=localhost:9092 acks=all

Record the records per second and the 99th percentile latency it prints. Throughput depends heavily on your node sizes, disks and network, so treat your own number as the result and note the hardware beside it.

What to commit

The Kafka, node pool, topic, Connect and connector manifests
apps/producer.py and apps/consumer.py
A short RESULTS.md with your measured throughput, the broker-failure outcome and a Grafana screenshot of lag and under-replicated partitions

Common mistakes

Following an older Strimzi tutorial that uses a zookeeper: section. Strimzi 0.46 and later reject it, so use KafkaNodePool and KRaft.
Setting replication.factor: 3 but leaving min.insync.replicas: 1. A single failure can then lose acknowledged writes. Set it to 2 and use acks=all.
Using kubectl exec to create topics. Declare them as KafkaTopic resources so they live in Git.
Hardcoding database passwords in manifests. Use Secrets and the config provider.
Reading lag without context. A short spike during a rebalance is normal. Sustained growth is the signal.

Found a mistake, an outdated step or a broken link?

Deploy Kafka on Kubernetes with Strimzi

Blueprint Walkthrough

Architecture Overview & Problem Statement

What you build, learn and can show

The problem this solves

Which versions this was written against

Prerequisites

Milestone 1: Install the Strimzi Operator

Milestone 2: Deploy a 3-Broker KRaft Kafka Cluster

Milestone 3: Create Topics with Proper Configuration

Milestone 4: Build Producer and Consumer Services

Milestone 5: Sink Orders into PostgreSQL with Kafka Connect

Deploy PostgreSQL

Deploy Kafka Connect with the JDBC plugin

Create the sink connector

Milestone 6: Monitor Kafka with Prometheus and Grafana

Validation & Testing

Broker failure test

Measure throughput on a separate topic

What to commit

Common mistakes