Deploy a Self-Healing Microservices Platform with Istio Service Mesh

Install Istio on EKS, deploy interconnected microservices, configure canary deployments, mutual TLS, circuit breaking, and visualise with Kiali.

Related Concepts & TermsProcess Deployment grep Grafana Circuit Breaking Canary Deployment

Domains & Technologies

Domains

ISTIOMONITORINGCI-CD

Technologies

KUBERNETES

Blueprint Walkthrough

Architecture Overview

This project deploys a complete service mesh using Istio on Kubernetes. A service mesh solves a fundamental problem — in a microservices architecture with 20 or 30 services, how do you control traffic between them, secure service-to-service communication, and understand what is happening inside your cluster without changing application code?

Istio answers all three questions by injecting a small proxy (Envoy) as a sidecar into every pod. All network traffic flows through these proxies, giving you traffic control, security, and observability for free regardless of what language your services are written in.

This is how PhonePe manages communication between their payment, wallet, and notification services — a service mesh that handles routing, retries, timeouts, and circuit breaking so application developers never have to write that logic themselves.

◈ DIAGRAM

    +------------------+
    |   Kiali UI       |  <- Visualises the entire service graph
    |   (Graph/Mesh)   |
    +------------------+
            |
    +------------------+
    |   Istiod         |  <- Control plane: configures all proxies
    |   (Control Plane)|  
    +------------------+
           / | \
          /  |  \
+---------+ +--------+ +----------+
| Service | | Service| | Service  |
|    A    | |   B    | |    C     |
| [Envoy] | |[Envoy] | | [Envoy]  |
+---------+ +--------+ +----------+
     ^           ^           ^
     |           |           |
     +-----mTLS--+-----------+
          All traffic encrypted

Problem Solved

Without a service mesh, each microservice team is responsible for implementing retry logic, timeouts, circuit breakers, mutual TLS, and distributed tracing individually. Teams copy-paste the same boilerplate across 20 services in 4 different programming languages. When something breaks, nobody knows which service is failing because there is no centralised view of traffic.

Istio moves all of this out of application code and into the infrastructure layer:

Traffic control — canary deployments, blue-green switching, weighted routing between service versions without redeploying anything.
Security — mutual TLS between every service automatically. Even if an attacker gets inside your cluster network, they cannot impersonate a legitimate service.
Observability — every request between services is automatically traced, measured, and visualised in Kiali without adding a single line of instrumentation code.
Resilience — circuit breakers that automatically stop sending traffic to a failing service, preventing cascade failures from taking down the entire platform.

Step-by-Step Implementation Guide

Step 1: Install Istio on Your Kubernetes Cluster

Istio has its own CLI called istioctl that handles installation and validation.

Bash

## Download the latest Istio release
curl -L https://istio.io/downloadIstio | sh -
 
## Move istioctl to your PATH
cd istio-1.20.0  # version may differ
export PATH=$PWD/bin:$PATH
echo 'export PATH=$HOME/istio-1.20.0/bin:$PATH' >> ~/.bashrc
 
## Verify istioctl is working
istioctl version
 
## Run pre-installation checks on your cluster
istioctl x precheck
## Expected: No issues found when checking the cluster
 
## Install Istio with the demo profile
## demo profile includes all components including tracing and Kiali
istioctl install --set profile=demo -y
 
## Verify all Istio components are running
kubectl get pods -n istio-system
## Expected: istiod, istio-ingressgateway, istio-egressgateway all Running
 
## Install Istio addons (Kiali, Prometheus, Grafana, Jaeger)
kubectl apply -f samples/addons/
kubectl rollout status deployment/kiali -n istio-system

Remember
The demo profile is for learning and includes everything. For production, use the default profile and add only the components you need. The demo profile installs Kiali, Jaeger, Prometheus, and Grafana which consume significant cluster resources.

Step 2: Deploy Three Interconnected Microservices

You will deploy an order service, payment service, and notification service. These three services call each other to simulate a real e-commerce transaction flow — similar to how Swiggy processes a food order.

Bash

## Create the application namespace and enable Istio sidecar injection
## The label tells Istio to automatically inject Envoy into every pod in this namespace
kubectl create namespace swiggy-demo
kubectl label namespace swiggy-demo istio-injection=enabled
 
## Verify the label is set
kubectl get namespace swiggy-demo --show-labels
## Expected: istio-injection=enabled
 
## Deploy the Order Service (v1 and v2 — we will use both for canary deployment later)
kubectl apply -n swiggy-demo -f - <<EOF
apiVersion: apps/v1
kind: Deployment
metadata:
  name: order-service-v1
spec:
  replicas: 2
  selector:
    matchLabels:
      app: order-service
      version: v1
  template:
    metadata:
      labels:
        app: order-service
        version: v1
    spec:
      containers:
      * name: order-service
        image: nginx:alpine
        ports:
        * containerPort: 80
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: order-service-v2
spec:
  replicas: 1
  selector:
    matchLabels:
      app: order-service
      version: v2
  template:
    metadata:
      labels:
        app: order-service
        version: v2
    spec:
      containers:
      * name: order-service
        image: nginx:alpine
        ports:
        * containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
  name: order-service
spec:
  selector:
    app: order-service
  ports:
  * port: 80
    name: http
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: payment-service
spec:
  replicas: 2
  selector:
    matchLabels:
      app: payment-service
      version: v1
  template:
    metadata:
      labels:
        app: payment-service
        version: v1
    spec:
      containers:
      * name: payment-service
        image: nginx:alpine
        ports:
        * containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
  name: payment-service
spec:
  selector:
    app: payment-service
  ports:
  * port: 80
    name: http
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: notification-service
spec:
  replicas: 1
  selector:
    matchLabels:
      app: notification-service
      version: v1
  template:
    metadata:
      labels:
        app: notification-service
        version: v1
    spec:
      containers:
      * name: notification-service
        image: nginx:alpine
        ports:
        * containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
  name: notification-service
spec:
  selector:
    app: notification-service
  ports:
  * port: 80
    name: http
EOF
 
## Verify all pods have 2 containers each (app + Envoy sidecar)
kubectl get pods -n swiggy-demo
## Expected: Each pod shows 2/2 READY — that second container is the Envoy proxy

Step 3: Configure Traffic Management with Virtual Services

Istio uses two custom resources to control traffic: DestinationRule defines the versions (subsets) of a service, and VirtualService defines the routing rules.

Bash

## Create DestinationRules — define v1 and v2 subsets for order-service
kubectl apply -n swiggy-demo -f - <<EOF
apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
  name: order-service
spec:
  host: order-service
  trafficPolicy:
    connectionPool:
      tcp:
        maxConnections: 100
      http:
        h2UpgradePolicy: UPGRADE
    outlierDetection:  # Circuit breaker configuration
      consecutive5xxErrors: 3      # Eject after 3 consecutive errors
      interval: 30s                # Check every 30 seconds
      baseEjectionTime: 30s        # Keep ejected for 30 seconds minimum
      maxEjectionPercent: 50       # Never eject more than 50% of endpoints
  subsets:
  * name: v1
    labels:
      version: v1
  * name: v2
    labels:
      version: v2
---
apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
  name: payment-service
spec:
  host: payment-service
  trafficPolicy:
    tls:
      mode: ISTIO_MUTUAL  # Enforce mutual TLS for payment service
  subsets:
  * name: v1
    labels:
      version: v1
EOF
 
## Create VirtualService — start with 100% traffic to v1 (stable release)
kubectl apply -n swiggy-demo -f - <<EOF
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: order-service
spec:
  hosts:
  * order-service
  http:
  * name: primary
    route:
    * destination:
        host: order-service
        subset: v1
      weight: 100  # 100% to v1 initially
    * destination:
        host: order-service
        subset: v2
      weight: 0    # 0% to v2 initially
EOF

Step 4: Implement Canary Deployment

A canary deployment gradually shifts traffic from the old version to the new version. You start at 5%, watch for errors, then increase to 20%, 50%, and finally 100%. If errors appear at any stage you route back to 0% instantly.

Bash

## Start the canary — send 10% of traffic to v2
kubectl apply -n swiggy-demo -f - <<EOF
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: order-service
spec:
  hosts:
  * order-service
  http:
  * route:
    * destination:
        host: order-service
        subset: v1
      weight: 90  # 90% to stable v1
    * destination:
        host: order-service
        subset: v2
      weight: 10  # 10% canary to v2
EOF
 
## Generate test traffic to see canary in action
for i in {1..100}; do
  kubectl exec -n swiggy-demo deployment/payment-service \
    -- curl -s http://order-service/healthz > /dev/null
done
 
## Watch the Kiali graph to see traffic split between v1 and v2
## Access Kiali:
kubectl port-forward -n istio-system svc/kiali 20001:20001
## Open http://localhost:20001 in your browser
## Navigate to Graph -> swiggy-demo namespace
## You should see animated traffic flowing between services
 
## If v2 looks healthy, increase to 50%
kubectl patch virtualservice order-service -n swiggy-demo \
  --type='json' \
  -p='[{"op": "replace", "path": "/spec/http/0/route/0/weight", "value": 50}, {"op": "replace", "path": "/spec/http/0/route/1/weight", "value": 50}]'
 
## Full rollout — 100% to v2
kubectl patch virtualservice order-service -n swiggy-demo \
  --type='json' \
  -p='[{"op": "replace", "path": "/spec/http/0/route/0/weight", "value": 0}, {"op": "replace", "path": "/spec/http/0/route/1/weight", "value": 100}]'
 
## Emergency rollback — instant switch back to v1 if problems appear
kubectl patch virtualservice order-service -n swiggy-demo \
  --type='json' \
  -p='[{"op": "replace", "path": "/spec/http/0/route/0/weight", "value": 100}, {"op": "replace", "path": "/spec/http/0/route/1/weight", "value": 0}]'

Tip
Canary deployment with Istio is entirely configuration-based — you never touch the actual Kubernetes deployments. The pods running v1 and v2 keep running throughout. Only the VirtualService routing weights change. This means rollback is instant — change the weight back to 100/0 and the rollback is complete in milliseconds.

Step 5: Enable Mutual TLS Across All Services

Mutual TLS (mTLS) means both the client and server verify each other's identity using certificates. Istio manages all certificates automatically through its built-in Certificate Authority.

Bash

## Check current mTLS status in the namespace
istioctl x describe service order-service.swiggy-demo
 
## Enable strict mTLS for the entire namespace
## This means ALL service-to-service communication must use mTLS
## Any pod without a valid Istio certificate cannot communicate
kubectl apply -f - <<EOF
apiVersion: security.istio.io/v1beta1
kind: PeerAuthentication
metadata:
  name: default
  namespace: swiggy-demo
spec:
  mtls:
    mode: STRICT  # Reject all non-mTLS traffic
EOF
 
## Verify mTLS is working — check the proxy config
istioctl proxy-config secret deployment/order-service-v1.swiggy-demo
## Expected: Shows the certificate details including expiry and SAN
 
## Try to access a service from outside the mesh (should fail)
kubectl run test-pod --image=curlimages/curl --rm -it \
  --restart=Never -- \
  curl http://order-service.swiggy-demo/healthz
## Expected: Connection refused — non-mesh clients cannot connect
 
## Try from inside the mesh (should succeed)
kubectl exec -n swiggy-demo deployment/payment-service \
  -- curl -s http://order-service/healthz
## Expected: Response from order-service — mesh clients can connect

Step 6: Test Circuit Breaking

The circuit breaker in the DestinationRule automatically stops sending traffic to a pod that is returning errors. This prevents a failing pod from dragging down the entire service.

Bash

## The circuit breaker is already configured in the DestinationRule:
## consecutive5xxErrors: 3 — eject after 3 consecutive 5xx errors
## interval: 30s — check window
## baseEjectionTime: 30s — how long to keep the pod ejected
 
## Simulate a failing pod by injecting a fault
kubectl apply -n swiggy-demo -f - <<EOF
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: payment-service-fault
spec:
  hosts:
  * payment-service
  http:
  * fault:
      abort:
        percentage:
          value: 100  # 100% of requests return 500
        httpStatus: 500
    route:
    * destination:
        host: payment-service
        subset: v1
EOF
 
## Send traffic to trigger the circuit breaker
for i in {1..20}; do
  kubectl exec -n swiggy-demo deployment/order-service-v1 \
    -- curl -s -o /dev/null -w "%{http_code}\n" http://payment-service/
done
## After 3 consecutive 500 errors, Istio ejects the endpoint
## Subsequent requests fail fast with 503 instead of waiting for timeout
 
## Check the circuit breaker stats in the Envoy proxy
kubectl exec -n swiggy-demo deployment/order-service-v1 \
  -c istio-proxy -- \
  pilot-agent request GET stats | grep outlier
## Look for: outlier_detection.ejections_active > 0
 
## Remove the fault injection to restore normal operation
kubectl delete virtualservice payment-service-fault -n swiggy-demo

Validation & Testing

Bash

## 1. Verify Istio is healthy
istioctl analyze -n swiggy-demo
## Expected: No validation issues found
 
## 2. Confirm all pods have Envoy sidecar (2 containers each)
kubectl get pods -n swiggy-demo
## Expected: All pods show 2/2 READY
 
## 3. Open Kiali and verify service graph shows all 3 services connected
kubectl port-forward -n istio-system svc/kiali 20001:20001
## http://localhost:20001 -> Graph -> swiggy-demo
 
## 4. Open Jaeger to see distributed traces
kubectl port-forward -n istio-system svc/tracing 16686:80
## http://localhost:16686 -> Select service -> Find Traces
 
## 5. Verify mTLS is enforced
istioctl x describe service payment-service.swiggy-demo | grep mTLS
## Expected: mTLS: STRICT
 
## 6. Run the canary from 0% to 100% in steps and verify traffic splits in Kiali
## Follow Milestone 4 steps and watch the animated percentages in the Kiali graph
 
## 7. Verify Grafana shows service metrics
kubectl port-forward -n istio-system svc/grafana 3000:3000
## http://localhost:3000 -> Istio Service Dashboard
## Should show request rate, error rate, latency for all 3 services
echo "Istio service mesh fully operational"

Videos & Guides

Video Guide

Istio Service Mesh Tutorial — Complete Crash Course

Complete Istio tutorial covering installation, traffic management, canary deployments, mutual TLS, circuit breaking, and observability with Kiali and Jaeger.

View Reference

DOCUMENTATION

Kiali — Service Mesh Observability

Official Kiali documentation for visualising and understanding your Istio service mesh — service graphs, traffic animation, configuration validation, and health indicators.

View Reference

DOCUMENTATION

Istio Official Documentation

Official Istio documentation covering all concepts including VirtualService, DestinationRule, PeerAuthentication, and the full traffic management API reference.

View Reference