Istio Service Mesh: Canary, mTLS, Circuit Breaking

Install Istio on Kubernetes, then run canary releases, strict mTLS and circuit breaking on a small microservice app and watch it live in Kiali.

Architecture Overview & Problem Statement

This project installs the Istio service mesh on a Kubernetes cluster and uses it to run weighted canary releases, enforce strict mutual TLS between services, and eject failing pods with circuit breaking, all without changing application code. A small set of stand-in microservices is deployed with Envoy sidecars, and Kiali, Prometheus and Jaeger show every request live. By the end you can prove each behaviour with a command that returns a number, not just a diagram.

A service mesh moves traffic control, security and observability out of every application and into the infrastructure. Istio injects an Envoy proxy as a sidecar next to each container, and a control plane called Istiod pushes routing and security configuration to all of those proxies. Without a mesh, every team re-implements retries, timeouts, TLS and tracing in its own language, and nobody has one view of how services talk to each other.

flowchart TD istiod[Istiod
control plane]:::ctl subgraph mesh[Namespace swiggy-demo] client[Client pod
plus Envoy]:::app order[Order v1 and v2
plus Envoy]:::app pay[Payment pods
plus Envoy]:::app end istiod -->|pushes config| client istiod -->|pushes config| order istiod -->|pushes config| pay client -->|mTLS| order client -->|mTLS| pay order --> prom[(Prometheus)]:::data pay --> prom prom --> kiali[Kiali graph]:::tool classDef ctl fill:#8B5CF6,color:#FFFFFF classDef app fill:#326CE5,color:#FFFFFF classDef data fill:#F97316,color:#FFFFFF classDef tool fill:#475569,color:#FFFFFF

What you will build

An Istio installation with Kiali, Prometheus, Grafana and Jaeger
Four stand-in services with sidecars, with two versions of the order service
A canary release that shifts traffic by exact percentages, plus a practiced rollback
Strict mTLS for the namespace, with a test that proves plain traffic is refused
Circuit breaking that ejects a failing pod, with a before and after error count

The services are small echo servers, not real business logic. That is deliberate: the mesh features work the same for any language, and echo servers let you tell versions apart and force failures on demand.

Prerequisites

A Kubernetes cluster with about 4 vCPU and 8 GB of free memory. kind, minikube or EKS all work
kubectl configured for that cluster
Basic knowledge of Deployments, Services and namespaces

Tip
The demo profile installs several tools and uses real memory. Use a throwaway cluster, and delete it when you finish.

Engineering Decision
This project is about resilience, meaning the mesh limits the damage of a bad release or a failing pod automatically. It does not repair broken code. Treat Istio as one layer of a reliable system, next to health probes, autoscaling and good alerts.

Milestone 1: Install Istio and the Observability Addons

This milestone installs the Istio control plane and the Kiali, Prometheus, Grafana and Jaeger addons.

istioctl is Istio's CLI and handles installation and validation. Pin the Istio version so your results match this guide. Use 1.22 or newer, because the networking.istio.io/v1 and security.istio.io/v1 APIs used below became stable in that release.

Bash

# Download a pinned Istio release (change the version to the current 1.22+ release you want)
curl -L https://istio.io/downloadIstio | ISTIO_VERSION=1.24.0 sh -
 
cd istio-1.24.0
export PATH=$PWD/bin:$PATH
 
istioctl version --remote=false
istioctl x precheck

Expected output from the precheck: No issues found when checking the cluster.

Bash

# The demo profile enables extra components, which suits learning
istioctl install --set profile=demo -y
 
kubectl get pods -n istio-system
 
# Addons: Kiali, Prometheus, Grafana, Jaeger
kubectl apply -f samples/addons/
kubectl rollout status deployment/kiali -n istio-system

Expected output: istiod and istio-ingressgateway are Running, and the Kiali rollout completes. The sample addons are for learning and are not production configurations.

Common Mistake
Applying samples/addons/ before Istio is healthy, or running it twice quickly and seeing errors about missing CRDs on the first pass. Wait until istiod is Running, then apply the addons. If a few resources fail, apply the same command again.

Milestone 2: Deploy the Demo Services with Sidecars

This milestone creates the namespace, labels it for sidecar injection, and deploys four services plus a client pod.

The label istio-injection=enabled tells Istio to inject an Envoy sidecar into every new pod in that namespace. Existing pods are not changed, so label the namespace before creating workloads.

Bash

kubectl create namespace swiggy-demo
kubectl label namespace swiggy-demo istio-injection=enabled
kubectl get namespace swiggy-demo --show-labels

Expected output includes istio-injection=enabled.

Create manifests/apps.yaml. Every echo pod prints its own name, so you can see which version answered. The payment service has two healthy pods and one pod that always returns HTTP 500, which you will use for circuit breaking later:

YAML

apiVersion: apps/v1
kind: Deployment
metadata:
  name: order-service-v1
spec:
  replicas: 2
  selector:
    matchLabels: {app: order-service, version: v1}
  template:
    metadata:
      labels: {app: order-service, version: v1}
    spec:
      containers:
        - name: app
          image: hashicorp/http-echo:1.0
          args: ["-listen=:5678", "-text=order-v1"]
          ports:
            - containerPort: 5678
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: order-service-v2
spec:
  replicas: 1
  selector:
    matchLabels: {app: order-service, version: v2}
  template:
    metadata:
      labels: {app: order-service, version: v2}
    spec:
      containers:
        - name: app
          image: hashicorp/http-echo:1.0
          args: ["-listen=:5678", "-text=order-v2"]
          ports:
            - containerPort: 5678
---
apiVersion: v1
kind: Service
metadata:
  name: order-service
spec:
  selector:
    app: order-service
  ports:
    - name: http
      port: 80
      targetPort: 5678
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: payment-service
spec:
  replicas: 2
  selector:
    matchLabels: {app: payment-service, version: healthy}
  template:
    metadata:
      labels: {app: payment-service, version: healthy}
    spec:
      containers:
        - name: app
          image: hashicorp/http-echo:1.0
          args: ["-listen=:5678", "-text=payment-ok"]
          ports:
            - containerPort: 5678
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: payment-service-faulty
spec:
  replicas: 1
  selector:
    matchLabels: {app: payment-service, version: faulty}
  template:
    metadata:
      labels: {app: payment-service, version: faulty}
    spec:
      containers:
        - name: app
          image: hashicorp/http-echo:1.0
          args: ["-listen=:5678", "-text=payment-broken", "-status-code=500"]
          ports:
            - containerPort: 5678
---
apiVersion: v1
kind: Service
metadata:
  name: payment-service
spec:
  selector:
    app: payment-service
  ports:
    - name: http
      port: 80
      targetPort: 5678
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: notification-service
spec:
  replicas: 1
  selector:
    matchLabels: {app: notification-service, version: v1}
  template:
    metadata:
      labels: {app: notification-service, version: v1}
    spec:
      containers:
        - name: app
          image: hashicorp/http-echo:1.0
          args: ["-listen=:5678", "-text=notification-v1"]
          ports:
            - containerPort: 5678
---
apiVersion: v1
kind: Service
metadata:
  name: notification-service
spec:
  selector:
    app: notification-service
  ports:
    - name: http
      port: 80
      targetPort: 5678
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: client
spec:
  replicas: 1
  selector:
    matchLabels: {app: client}
  template:
    metadata:
      labels: {app: client}
    spec:
      containers:
        - name: client
          image: curlimages/curl:8.10.1
          command: ["sleep", "infinity"]

Bash

kubectl apply -n swiggy-demo -f manifests/apps.yaml
kubectl rollout status deployment/client -n swiggy-demo
kubectl get pods -n swiggy-demo

Expected output: every pod shows 2/2 under READY. The second container is the Envoy sidecar.

Before adding any routing rules, see how traffic splits by default. The order service has 2 v1 pods and 1 v2 pod:

Bash

kubectl exec -n swiggy-demo deploy/client -c client -- \
  sh -c 'for i in $(seq 1 90); do curl -s http://order-service/; done' | sort | uniq -c

Expected output: roughly 60 responses of order-v1 and 30 of order-v2. With no rules, traffic follows the pod count, so you cannot give v2 exactly 10%. That is the problem the next milestone solves.

Common Mistake
Pods showing 1/1 instead of 2/2. The namespace was labelled after the pods were created. Restart them with kubectl rollout restart deployment -n swiggy-demo so the sidecars are injected.

Milestone 3: Route Traffic with DestinationRules and VirtualServices

This milestone defines version subsets and a routing rule that sends all order traffic to v1, ready for a canary.

Istio uses two resources together. A DestinationRule names the subsets of a service, here v1 and v2, by pod label. A VirtualService decides how requests are routed across those subsets. Weights in the VirtualService are exact percentages, no matter how many pods each version has.

Create manifests/order-routing.yaml:

YAML

apiVersion: networking.istio.io/v1
kind: DestinationRule
metadata:
  name: order-service
spec:
  host: order-service
  subsets:
    - name: v1
      labels:
        version: v1
    - name: v2
      labels:
        version: v2
---
apiVersion: networking.istio.io/v1
kind: VirtualService
metadata:
  name: order-service
spec:
  hosts:
    - order-service
  http:
    - route:
        - destination:
            host: order-service
            subset: v1
          weight: 100
        - destination:
            host: order-service
            subset: v2
          weight: 0

Bash

kubectl apply -n swiggy-demo -f manifests/order-routing.yaml
istioctl analyze -n swiggy-demo

Expected output from analyze: No validation issues found. Rerun the 90-request loop from the previous milestone and confirm every response is now order-v1.

Tip
istioctl analyze catches the most common mistakes, such as a VirtualService pointing at a subset that does not exist. Run it after every change.

Common Mistake
A VirtualService weight total that does not add to 100, or a subset name that has no matching pod labels. Traffic then fails with 503 errors. Check subset labels against kubectl get pods --show-labels.

Milestone 4: Run a Canary Release and Roll Back

This milestone shifts a small share of traffic to v2, measures the split, then practises a rollback after a bad release.

A canary release sends a small percentage of real traffic to the new version first. You watch error rates, and either increase the percentage or return to zero. Because only routing weights change, the pods stay running and a rollback takes effect as soon as the proxies receive the new config.

Create set-canary.sh, a helper that applies exact weights:

Bash

#!/usr/bin/env bash
set -euo pipefail
V1=${1:?usage: set-canary.sh V1_WEIGHT V2_WEIGHT}
V2=${2:?usage: set-canary.sh V1_WEIGHT V2_WEIGHT}
 
kubectl apply -n swiggy-demo -f - <<EOF
apiVersion: networking.istio.io/v1
kind: VirtualService
metadata:
  name: order-service
spec:
  hosts:
    - order-service
  http:
    - route:
        - destination:
            host: order-service
            subset: v1
          weight: ${V1}
        - destination:
            host: order-service
            subset: v2
          weight: ${V2}
EOF

Bash

chmod +x set-canary.sh
./set-canary.sh 90 10
 
kubectl exec -n swiggy-demo deploy/client -c client -- \
  sh -c 'for i in $(seq 1 200); do curl -s http://order-service/; done' | sort | uniq -c

Expected output: close to 180 order-v1 and 20 order-v2. The split is random, so expect small differences between runs.

Watch it in Kiali

In one terminal, keep steady traffic flowing so the graph has data:

Bash

kubectl exec -n swiggy-demo deploy/client -c client -- sh -c '
while true; do
  curl -s http://order-service/ > /dev/null
  curl -s http://notification-service/ > /dev/null
  curl -s http://payment-service/ > /dev/null
  sleep 0.5
done'

In a second terminal, open Kiali:

Bash

kubectl port-forward -n istio-system svc/kiali 20001:20001

Open http://localhost:20001, choose Graph, select the swiggy-demo namespace, and enable the Versioned app graph. You should see the client sending traffic to order-service v1 and v2 in the weights you set.

Practise a bad canary and a rollback

Make v2 fail on purpose, send it 20% of traffic, and measure the damage:

Bash

kubectl patch deployment order-service-v2 -n swiggy-demo --type=json -p='[
  {"op":"replace","path":"/spec/template/spec/containers/0/args",
   "value":["-listen=:5678","-text=order-v2","-status-code=500"]}]'
kubectl rollout status deployment/order-service-v2 -n swiggy-demo
 
./set-canary.sh 80 20
 
kubectl exec -n swiggy-demo deploy/client -c client -- \
  sh -c 'for i in $(seq 1 100); do curl -s -o /dev/null -w "%{http_code}\n" http://order-service/; done' \
  | sort | uniq -c

Expected output: about 80 responses of 200 and about 20 of 500. Now roll back and measure again:

Bash

./set-canary.sh 100 0
 
kubectl exec -n swiggy-demo deploy/client -c client -- \
  sh -c 'for i in $(seq 1 100); do curl -s -o /dev/null -w "%{http_code}\n" http://order-service/; done' \
  | sort | uniq -c

Expected output: 100 responses of 200. The error rate returned to zero without touching any Deployment.

Common Mistake
Promoting a canary on a hunch. Decide the promotion rule before the release, for example "error rate under 1% for 10 minutes", and read the numbers from Kiali or Prometheus. Gut feeling is how a bad version reaches 100%.

Milestone 5: Enforce Strict Mutual TLS

This milestone turns on strict mTLS for the namespace and proves that plain, non-mesh clients are refused.

Mutual TLS means both sides of a connection present certificates and verify each other. Istiod acts as a certificate authority and rotates short-lived certificates for every workload automatically. By default Istio runs in PERMISSIVE mode, which accepts both mTLS and plain traffic so migrations are safe. STRICT accepts only mTLS.

First confirm that plain traffic works today. This pod runs in the default namespace, which has no sidecar injection:

Bash

kubectl run plain-client -n default --image=curlimages/curl:8.10.1 --rm -it \
  --restart=Never --command -- \
  curl -s -m 5 -o /dev/null -w "%{http_code}\n" http://order-service.swiggy-demo/

Expected output: 200. Now enforce strict mTLS. Create manifests/peer-auth.yaml:

YAML

apiVersion: security.istio.io/v1
kind: PeerAuthentication
metadata:
  name: default
  namespace: swiggy-demo
spec:
  mtls:
    mode: STRICT

Bash

kubectl apply -f manifests/peer-auth.yaml
kubectl get peerauthentication -n swiggy-demo

Rerun the plain client test:

Bash

kubectl run plain-client -n default --image=curlimages/curl:8.10.1 --rm -it \
  --restart=Never --command -- \
  curl -s -m 5 -o /dev/null -w "%{http_code}\n" http://order-service.swiggy-demo/

Expected output: the request fails with a connection reset (curl exit code 56, Recv failure: Connection reset by peer). The sidecar refuses plaintext. Now confirm that mesh clients still work and that certificates exist:

Bash

kubectl exec -n swiggy-demo deploy/client -c client -- curl -s http://order-service/
istioctl proxy-config secret deploy/order-service-v1 -n swiggy-demo

Expected output: the first command prints order-v1. The second lists a default certificate and a ROOTCA entry with validity dates. In Kiali, turn on the Security display option and look for padlock icons on the graph edges.

Security
mTLS proves who is calling and encrypts the traffic, but it does not decide who is allowed to call whom. Add an AuthorizationPolicy next, for example so only the order service can call the payment service. Without it, any workload in the mesh can reach any other.

Common Mistake
Switching to STRICT before every workload has a sidecar. Anything without one, such as a legacy pod or a job in another namespace, loses connectivity at once. Stay in PERMISSIVE, check Kiali for non-mTLS edges, and only then enforce STRICT.

Milestone 6: Add Circuit Breaking

This milestone makes Istio eject the payment pod that keeps returning HTTP 500, and measures how many errors clients see before and after.

A circuit breaker stops sending traffic to an endpoint that keeps failing, so one bad pod does not keep hurting users. In Istio this is called outlier detection, and it works on pods, not on whole services. Do not test it with a VirtualService fault injection that aborts requests, because that fault happens before load balancing and never marks a pod as unhealthy. A pod that really returns 500 does.

stateDiagram-v2 [*] --> InPool InPool --> Ejected: 3 consecutive 5xx Ejected --> InPool: ejection time ends

Measure the baseline

The payment service has two healthy pods and one faulty pod. Without protection, about a third of calls fail:

Bash

kubectl exec -n swiggy-demo deploy/client -c client -- \
  sh -c 'for i in $(seq 1 60); do curl -s -o /dev/null -w "%{http_code}\n" http://payment-service/; done' \
  | sort | uniq -c

Expected output: about 40 responses of 200 and about 20 of 500.

Turn on outlier detection

Create manifests/payment-circuit-breaker.yaml:

YAML

apiVersion: networking.istio.io/v1
kind: DestinationRule
metadata:
  name: payment-service
spec:
  host: payment-service
  trafficPolicy:
    connectionPool:
      tcp:
        maxConnections: 100
    outlierDetection:
      consecutive5xxErrors: 3
      interval: 5s
      baseEjectionTime: 60s
      maxEjectionPercent: 50

Bash

kubectl apply -n swiggy-demo -f manifests/payment-circuit-breaker.yaml
 
kubectl exec -n swiggy-demo deploy/client -c client -- \
  sh -c 'for i in $(seq 1 60); do curl -s -o /dev/null -w "%{http_code}\n" http://payment-service/; done' \
  | sort | uniq -c

Expected output: only a few 500 responses, about three, while the proxy learns that the pod is failing, and then 200 for the rest. Confirm which pod was ejected:

Bash

istioctl proxy-config endpoint deploy/client.swiggy-demo \
  --cluster "outbound|80||payment-service.swiggy-demo.svc.cluster.local"

Expected output: two endpoints show HEALTHY and one shows FAILED_OUTLIER_CHECK. After baseEjectionTime, Istio lets the pod back in, and if it fails again it is ejected for longer each time.

Tip
maxEjectionPercent: 50 stops the mesh from ejecting too many pods at once. If every pod fails, you want the service degraded, not completely empty. Keep this value below 100.

Common Mistake
Expecting outlier detection to work with a single replica or to hide errors from users completely. The first few failing requests still reach the client, and with only one pod there is nowhere else to send traffic. Pair circuit breaking with at least two replicas and with retries for transient errors.

Stretch: add retries

Retries hide short blips, and they pair well with outlier detection. Try this VirtualService for the payment service and rerun the 60-request test. Compare the error count with the numbers above:

YAML

apiVersion: networking.istio.io/v1
kind: VirtualService
metadata:
  name: payment-service
spec:
  hosts:
    - payment-service
  http:
    - route:
        - destination:
            host: payment-service
      retries:
        attempts: 2
        perTryTimeout: 2s
        retryOn: 5xx

Only retry operations that are safe to repeat. Retrying a payment charge that is not idempotent can charge a customer twice.

Validation & Testing

These checks prove each feature works with numbers. Keep the background traffic loop from Milestone 4 running if you want a live Kiali view.

1. Istio and configuration are healthy

Bash

istioctl analyze -n swiggy-demo
kubectl get pods -n swiggy-demo

Expected output: No validation issues found, and every pod shows 2/2 ready.

2. Canary split is accurate

Bash

kubectl patch deployment order-service-v2 -n swiggy-demo --type=json -p='[
  {"op":"replace","path":"/spec/template/spec/containers/0/args",
   "value":["-listen=:5678","-text=order-v2"]}]'
kubectl rollout status deployment/order-service-v2 -n swiggy-demo
 
./set-canary.sh 90 10
kubectl exec -n swiggy-demo deploy/client -c client -- \
  sh -c 'for i in $(seq 1 200); do curl -s http://order-service/; done' | sort | uniq -c

Expected output: roughly 180 order-v1 and 20 order-v2.

3. Rollback is clean

Bash

./set-canary.sh 100 0
kubectl exec -n swiggy-demo deploy/client -c client -- \
  sh -c 'for i in $(seq 1 100); do curl -s http://order-service/; done' | sort | uniq -c

Expected output: 100 order-v1.

4. Strict mTLS is enforced

Bash

kubectl get peerauthentication -n swiggy-demo
kubectl run plain-client -n default --image=curlimages/curl:8.10.1 --rm -it \
  --restart=Never --command -- \
  curl -s -m 5 -o /dev/null -w "%{http_code}\n" http://order-service.swiggy-demo/

Expected output: the mode shows STRICT, and the plain client fails with a connection reset. A client inside the mesh still receives order-v1.

5. Circuit breaking ejects the failing pod

Bash

istioctl proxy-config endpoint deploy/client.swiggy-demo \
  --cluster "outbound|80||payment-service.swiggy-demo.svc.cluster.local"

Expected output: one endpoint reports FAILED_OUTLIER_CHECK while the two healthy pods report HEALTHY.

6. Observability shows the traffic

Bash

kubectl port-forward -n istio-system svc/kiali 20001:20001
kubectl port-forward -n istio-system svc/tracing 16686:80
kubectl port-forward -n istio-system svc/grafana 3000:3000

Expected output: Kiali at http://localhost:20001 shows the versioned service graph with padlocks. Jaeger at http://localhost:16686 lists traces for your services. Grafana at http://localhost:3000 has an Istio Service Dashboard with request rate, error rate and latency. Run each port-forward in its own terminal.

7. Clean up

Bash

kubectl delete namespace swiggy-demo
istioctl uninstall --purge -y
kubectl delete namespace istio-system

Command reference

Command	What it shows
`istioctl analyze -n NAMESPACE`	Configuration mistakes in a namespace
`istioctl proxy-config endpoint POD --cluster NAME`	Endpoint health, including ejected pods
`istioctl proxy-config secret POD`	The workload's mTLS certificates
`kubectl get peerauthentication -n NAMESPACE`	Which mTLS mode is enforced
`kubectl get virtualservice,destinationrule -n NAMESPACE`	Active routing and traffic rules

Portfolio deliverable

Keep your manifests/ folder, the three before and after counts (canary split, bad canary versus rollback, circuit breaker errors), and a Kiali screenshot showing versioned traffic with padlocks. Record your own measured numbers, since splits and ejection timing vary from run to run.

Found a mistake, an outdated step or a broken link?

Istio Service Mesh: Canary, mTLS, Circuit Breaking

Blueprint Walkthrough

Architecture Overview & Problem Statement

What you will build

Prerequisites

Milestone 1: Install Istio and the Observability Addons

Milestone 2: Deploy the Demo Services with Sidecars

Milestone 3: Route Traffic with DestinationRules and VirtualServices

Milestone 4: Run a Canary Release and Roll Back

Watch it in Kiali

Practise a bad canary and a rollback

Milestone 5: Enforce Strict Mutual TLS

Milestone 6: Add Circuit Breaking

Measure the baseline

Turn on outlier detection

Stretch: add retries

Validation & Testing

1. Istio and configuration are healthy

2. Canary split is accurate

3. Rollback is clean

4. Strict mTLS is enforced

5. Circuit breaking ejects the failing pod

6. Observability shows the traffic

7. Clean up

Command reference

Portfolio deliverable