## Understanding the Problem Service Mesh Solves You have 40 microservices running on Kubernetes. Payments, orders, inventory, notifications, user profiles - all talking to each other over the network. Everything works fine in staging with 5 services. In production with 40, things get messy fast. The order service calls the payments service. The payments service is slow - maybe a database is struggling, maybe there is a bug in a new deployment. The order service waits. Then more requests pile up. The order service runs out of threads waiting for payments to respond. Now the order service is also slow. Every service that calls orders starts waiting too. Within minutes, a problem in one service has taken down the entire system. This is called a **cascading failure**. Kubernetes gives you a way to run services. It does not give you any of the following: * Automatic retries when a service call fails * Encrypted communication between services inside the cluster * The ability to send 5% of traffic to a new version before rolling it out fully * A dashboard showing which services are slow and why * Automatic circuit breaking that stops calls to a failing service You could add all of this inside every service yourself. But that means writing the same networking logic in every service, in every programming language your teams use. When you need to change that logic, you change 40 services. This is where a service mesh steps in. ### What raw Kubernetes networking gives you Kubernetes networking handles one thing well - getting packets from one pod to another. Every pod gets an IP address. Services get a stable DNS name. Traffic routes from a Service to its matching pods through kube-proxy. That is where Kubernetes stops. It does not know if the response was an error. It does not retry. It does not encrypt. It does not measure latency per service. It treats all traffic as equal regardless of which version a pod is running. For two or three services this is fine. For 20 or 40, it is not enough. ### Where service mesh fits in the platform engineering stack If you have already set up Kubernetes deployments, you are adding a service mesh as a layer on top - it does not replace anything. Your pods keep running. Your services keep working. The mesh adds capabilities underneath your application code without requiring any changes to it. Your application code | Service Mesh (Istio or Linkerd) | Kubernetes networking (kube-proxy, CNI) | Physical or cloud network Everything your app does still works. The mesh intercepts traffic, applies policies, and collects data at the layer below your code. --- ## Understanding the Sidecar Pattern The key mechanism behind every service mesh is the **sidecar proxy**. Understanding this one pattern makes everything else click. Think about a secretary who sits outside every meeting room. Every letter that goes into the room passes through the secretary. Every letter that comes out passes through the secretary too. The people inside the room do not know the secretary is there - they just send and receive letters as normal. But the secretary logs everything, checks credentials, can hold back letters if the room is too busy, and can route urgent letters to a backup room if the main one is unavailable. That is exactly what a sidecar proxy does for your pods. ### How the sidecar gets attached to every pod When you install a service mesh on a cluster, it modifies how pods are created. It watches for new pods and automatically injects a second container into each one. This second container is the sidecar proxy - Envoy in Istio's case, a lightweight Rust-based proxy in Linkerd's case. Pod (before mesh) Pod (after mesh) +------------------+ +------------------+ | Your app | | Your app | | container | | container | +------------------+ +------------------+ | Sidecar proxy | | (auto-injected)| +------------------+ The sidecar and your app container share the same network namespace. This means the sidecar can intercept all incoming and outgoing traffic using iptables rules - without your app knowing. Your app thinks it is talking directly to the payments service. In reality, it is talking to its own sidecar, which talks to the payments service's sidecar, which then delivers the traffic to the payments app. ### The control plane and data plane A service mesh has two parts. Once you know what they do, reading Istio and Linkerd documentation becomes much clearer. **The data plane** is all the sidecars running alongside your pods. This is where actual traffic flows. Sidecars intercept, forward, encrypt, and collect metrics on every request. **The control plane** is the central brain. It watches the cluster for service changes, calculates routing rules, and pushes configuration down to all the sidecars. In Istio this is called `istiod`. In Linkerd this is called the control plane. Sidecars connect to the control plane at startup and receive their configuration from it. The control plane never sees your actual traffic. It only pushes config. All traffic handling happens in the data plane sidecars. --- ## Implementing mTLS Between Services **mTLS** stands for mutual TLS. Regular TLS is one-sided - your browser verifies the server's identity, but the server does not verify yours. Mutual TLS means both sides verify each other's identity. Every service proves who it is before communication begins. Without mTLS inside your cluster, traffic between pods is plain text. Anyone with access to the cluster network - a compromised pod, a malicious container, a misconfigured network policy - can read that traffic. At Zerodha, where internal services pass financial transaction data between them, unencrypted internal traffic is a serious compliance and security risk. ### How mTLS works inside the mesh without touching your code This is the part that seems like magic until you understand the sidecar pattern. Your application code opens a plain HTTP connection to the payments service. Your sidecar intercepts that connection. The sidecar holds a certificate issued by the mesh's certificate authority. The payments service's sidecar also holds a certificate from the same CA. The two sidecars perform a TLS handshake between themselves, verifying each other's identities using those certificates. The encrypted tunnel is between the sidecars. Your app code never changes. It still opens plain HTTP. The encryption happens transparently below it. ```yaml ## Enable mTLS for the entire production namespace apiVersion: security.istio.io/v1beta1 kind: PeerAuthentication metadata: name: default namespace: production spec: mtls: mode: STRICT ## only encrypted traffic allowed, plain rejected ``` > **Note:** `STRICT` mode means any service without a sidecar cannot communicate > in this namespace. Use `PERMISSIVE` first when migrating an existing cluster > so services without sidecars still work while you roll out injection gradually. > 📌 **Remember:** mTLS certificates in Istio rotate automatically every 24 hours. > You do not manage them manually. The control plane handles issuance and rotation. ### Verifying mTLS is active ```bash ## Check that mTLS is enforced between two services istioctl x authz check \ $(kubectl get pod -l app=orders -n production \ -o jsonpath='{.items[0].metadata.name}') \ -n production ``` ```text ACTION AuthorizationPolicy RULES ALLOW - - ``` > **Note:** If mTLS is working, traffic between services will show as encrypted > in the Kiali dashboard (lock icon on service graph edges). --- ## Implementing Canary Releases with Traffic Splitting A **canary release** is a deployment strategy where you send a small percentage of real traffic to a new version before fully rolling it out. The name comes from the old mining practice of sending a canary into a tunnel first - if it survives, it is safe for people. At Swiggy, releasing a new version of the restaurant service to 100% of users at once is risky. If there is a bug, every user is affected. With a canary, you send 5% of traffic to v2, watch error rates and latency for 30 minutes, and only promote to 100% if metrics look healthy. ### Without a service mesh Without a mesh, doing traffic splitting requires running two separate Kubernetes Deployments and adjusting replica counts to approximate percentages. Want 5% to v2? Run 1 pod of v2 and 19 pods of v1. This is imprecise, expensive, and impossible to control below the granularity of your pod count. ### With Istio VirtualService With Istio, you define traffic weights explicitly and precisely. It does not matter how many pods each version has. ```yaml ## Route 95% of traffic to v1, 5% to v2 of the restaurant service apiVersion: networking.istio.io/v1alpha3 kind: VirtualService metadata: name: restaurant-service namespace: production spec: hosts: - restaurant-service ## the Kubernetes Service name http: - route: - destination: host: restaurant-service subset: v1 ## defined in DestinationRule below weight: 95 - destination: host: restaurant-service subset: v2 weight: 5 ``` ```yaml ## Define which pods belong to v1 and which to v2 apiVersion: networking.istio.io/v1alpha3 kind: DestinationRule metadata: name: restaurant-service namespace: production spec: host: restaurant-service subsets: - name: v1 labels: version: "v1" ## pods with this label get v1 traffic - name: v2 labels: version: "v2" ## pods with this label get v2 traffic ``` > **Note:** `VirtualService` controls how traffic is routed. `DestinationRule` > controls which pods belong to each subset. You need both for traffic splitting to work. To promote v2 after validation, change the weights to `weight: 0` and `weight: 100`. To roll back, delete the VirtualService and all traffic returns to the default Service routing. ### Header-based routing for internal testing ```yaml ## Send traffic to v2 only when request has header x-canary: true ## For internal QA teams to test v2 without affecting real users http: - match: - headers: x-canary: exact: "true" route: - destination: host: restaurant-service subset: v2 - route: - destination: host: restaurant-service subset: v1 ``` --- ## Implementing Circuit Breaking **Circuit breaking** is named after electrical circuit breakers. When too much current flows through a circuit, the breaker trips and stops the flow before the wiring catches fire. In software, a circuit breaker stops sending requests to a service that is failing - before those failed requests pile up and take down the calling service too. Without circuit breaking, if the payments service is down and orders keeps retrying, orders fills up its thread pool waiting for responses. Orders becomes slow. Notification service, which calls orders, also fills up. Within minutes, a payments failure has cascaded into a full system outage. With circuit breaking, after a configured number of failures, the mesh stops forwarding requests to payments entirely and returns an error immediately. The calling service can handle that fast error gracefully instead of hanging. When payments recovers, the circuit closes and traffic resumes. ### Configuring circuit breaking in Istio ```yaml ## Trip the circuit breaker after 5 consecutive errors ## or if connection pool is exhausted apiVersion: networking.istio.io/v1alpha3 kind: DestinationRule metadata: name: payments-circuit-breaker namespace: production spec: host: payments-service trafficPolicy: connectionPool: http: http1MaxPendingRequests: 100 ## max queued before rejection maxRequestsPerConnection: 10 ## max per connection outlierDetection: consecutive5xxErrors: 5 ## trip after 5 consecutive 5xx interval: 10s ## check error rate every 10s baseEjectionTime: 30s ## eject failing pod for 30s maxEjectionPercent: 50 ## max 50% of pods ejectable ``` > 📌 **Remember:** Circuit breaking in Istio works at the pod level, not the > service level. If one pod in the payments Deployment is failing, that specific > pod gets ejected. Healthy pods in the same Deployment keep receiving traffic. > This is called **outlier detection**. > 🔴 **Common Mistake:** Setting `maxEjectionPercent: 100` means Istio can eject > all pods if they all start returning errors. This makes your service completely > unreachable. Keep it at 50% or lower so at least half your pods always remain > in rotation. --- ## Observability with Kiali and Metrics The third major benefit of a service mesh is observability you get for free. Because every request passes through a sidecar, the mesh can measure latency, error rates, and request volume for every service-to-service call - without any instrumentation in your application code. **Kiali** is the standard observability dashboard for Istio. It shows a live graph of your services with traffic flowing between them. Each edge in the graph shows request rate, error rate, and latency. Each node shows health status. ### What you can see in Kiali * Which services are calling which other services (and which you did not know about) * Where in the call chain a slow response is originating * Which version of a service is receiving traffic during a canary rollout * Which circuit breakers have tripped * Whether mTLS is active on each connection (shown as a lock icon) ### Metrics automatically collected by the sidecar ```bash ## Istio sidecars expose Prometheus metrics at port 15090 ## These are scraped automatically if you have Prometheus installed ## Example: check raw metrics from an orders pod sidecar kubectl exec -n production \ $(kubectl get pod -l app=orders -n production \ -o jsonpath='{.items[0].metadata.name}') \ -c istio-proxy -- \ curl -s localhost:15090/metrics | grep istio_requests ``` Key metrics the mesh provides out of the box: | Metric | What it measures | |:---|:---| | `istio_requests_total` | Total request count, labelled by source, destination, status code | | `istio_request_duration_milliseconds` | Latency histogram per service pair | | `istio_tcp_connections_opened_total` | TCP connection rate for non-HTTP traffic | > 💡 **Tip:** Install the Istio addons bundle to get Kiali, Prometheus, Grafana, > and Jaeger all preconfigured together. One command: > `kubectl apply -f https://raw.githubusercontent.com/istio/istio/release-1.20/samples/addons/kiali.yaml` --- ## Choosing Between Istio and Linkerd The two most widely used service meshes are Istio and Linkerd. They solve the same core problems but make very different trade-offs. Choosing the wrong one adds unnecessary operational complexity. | Dimension | Istio | Linkerd | |:---|:---|:---| | Complexity | High - many CRDs, many config options | Low - focused feature set, fewer concepts | | Resource overhead | Higher - Envoy proxy uses more CPU and RAM | Lower - Rust proxy is very lightweight | | mTLS | Yes, automatic | Yes, automatic | | Traffic splitting | Yes, via VirtualService | Yes, via HTTPRoute | | Circuit breaking | Yes, via DestinationRule | Partial - retry budgets, no outlier detection | | Observability | Kiali, Jaeger, Prometheus built in | Viz dashboard built in, lightweight | | Multi-cluster | Yes, strong support | Yes, good support | | Learning curve | Steep - Envoy concepts required | Gentle - simpler mental model | | Best for | Large teams, complex routing needs, compliance requirements | Teams wanting fast setup, lower overhead, simpler ops | | Protocol support | HTTP/1, HTTP/2, gRPC, TCP | HTTP/1, HTTP/2, gRPC, TCP | ### When to choose Istio Pick Istio when your team needs advanced traffic management - fault injection for chaos engineering, complex header-based routing, WebAssembly extensions for custom proxy behaviour, or strong multi-cluster federation. Istio is also the default choice when teams are already investing in the CNCF ecosystem and need the widest community support. The trade-off is operational complexity. Istio has over 50 Custom Resource Definitions. A new engineer joining the team will need time to understand VirtualServices, DestinationRules, Gateways, PeerAuthentication, AuthorizationPolicies, and ServiceEntries before they can safely make changes. ### When to choose Linkerd Pick Linkerd when you want the core mesh capabilities - mTLS, retries, observability, traffic splitting - with the fastest installation path and lowest operational overhead. Linkerd's data plane proxy is written in Rust and uses significantly less CPU and memory than Envoy. For teams running on smaller clusters or with strict resource budgets, this matters. Linkerd does not support some of the more advanced Istio features like fault injection or WebAssembly extensions. If your requirements are: secure service-to-service communication, automatic retries, and a service topology dashboard, Linkerd delivers all of this with far less configuration surface area. > 📌 **Remember:** Both meshes are CNCF projects. Linkerd is a graduated CNCF > project. Istio became a CNCF graduated project in 2023. Either is a safe, > production-grade choice. The decision is about operational trade-offs, not maturity. --- ## Hands-On - Installing Istio and Running a Canary Release This walkthrough installs Istio on a local Kubernetes cluster, deploys two versions of a service, and performs a live canary release by shifting traffic between versions. Prerequisites: `kubectl` configured against a running cluster (Minikube, Kind, or cloud), and `istioctl` downloaded from `istio.io/downloadIstio`. 1. Download and install the `istioctl` CLI: ```bash ## Download Istio 1.20 (latest stable at time of writing) curl -L https://istio.io/downloadIstio | ISTIO_VERSION=1.20.0 sh - ## Add istioctl to your PATH for this session export PATH=$PWD/istio-1.20.0/bin:$PATH ## Verify the CLI is available istioctl version ``` 2. Install Istio onto your cluster using the demo profile: ```bash ## demo profile includes addons and is suitable for learning ## production profile is leaner - use it for real clusters istioctl install --set profile=demo -y ## Verify all Istio components are running kubectl get pods -n istio-system ``` ```text NAME READY STATUS RESTARTS istio-ingressgateway-xxx 1/1 Running 0 istiod-xxx 1/1 Running 0 ``` 3. Enable automatic sidecar injection for your namespace: ```bash ## Label namespace so Istio injects sidecars automatically ## Every new pod created in this namespace will get a sidecar kubectl create namespace production kubectl label namespace production istio-injection=enabled ## Verify the label is applied kubectl get namespace production --show-labels ``` 4. Deploy two versions of an orders service: ```bash ## Save this as orders-v1.yaml cat < **Note:** `2/2` means both containers in the pod are running. The second container > is the Istio sidecar proxy. If you see `1/1`, sidecar injection did not work - > check that the namespace has the `istio-injection=enabled` label. 6. Apply DestinationRule to define v1 and v2 subsets: ```bash cat <