Zero-Trust Security with Vault and Istio mTLS

Give pods short-lived Vault database credentials, enforce Istio mTLS and identity-based access, then prove it holds with chaos experiments.

Architecture Overview & Problem Statement

This project builds a zero-trust setup on a local Kubernetes cluster with two tools: HashiCorp Vault hands each pod short-lived database credentials that expire on their own, and Istio encrypts and authenticates every service-to-service call with mutual TLS (mTLS), so only named workloads can talk to each other. You then run small chaos experiments to prove the controls hold when pods are killed and credentials are revoked.

The problem it solves is trust by location. A static database password in a Secret works for anyone who finds it, forever, and plain traffic inside the cluster trusts any pod that can reach the port. Here, every credential expires within minutes and every connection proves who is calling.

Key ideas

Dynamic secrets: Vault creates a new database user on demand with a time limit and deletes it at expiry. A leaked credential stops working by itself.
mTLS: both sides of a connection present certificates, so the server knows exactly which workload is calling, not just which IP.
Authorization policy: after identity is proven, a policy decides which identities may call which service. Anything not allowed is denied.

Scope and prerequisites

This project uses its own fresh kind cluster, so you do not need the cluster from the Kubernetes hardening project. Infrastructure-as-code scanning and a findings dashboard are covered in the pipeline projects and are not repeated here.

Docker, kubectl, Helm and kind installed (a two-node kind cluster is enough)
About 8 GB of free memory for Istio, Vault and Postgres together
Familiarity with pods, services and namespaces

Security
Vault runs in dev mode here. Dev mode stores everything in memory, uses a known root token and has no TLS. It is perfect for learning and unacceptable for real use. A production Vault needs persistent storage, high availability, auto-unseal and TLS.

Tip
The versions used are Istio 1.31.1 and the Vault Helm chart 0.34.1. If you use newer ones, read the release notes first because Istio and Vault change defaults between releases.

Milestone 1: Create the Cluster and Install Istio and Vault

This milestone creates the cluster and installs the two platforms. Istio goes in with the minimal profile, which installs only the control plane (istiod), because you only need sidecar mTLS, not an ingress gateway.

Create the cluster and namespaces

Bash

kind create cluster --name zerotrust
kubectl create namespace vault
kubectl create namespace data
kubectl create namespace legacy
kubectl create namespace payments
kubectl label namespace payments istio-injection=enabled

Only payments is labelled for sidecar injection. vault, data and legacy stay outside the mesh on purpose: the first two keep Vault and the database simple, and legacy plays the part of a workload with no sidecar.

Install Istio

Bash

curl -L https://istio.io/downloadIstio | ISTIO_VERSION=1.31.1 TARGET_ARCH=x86_64 sh -
export PATH="$PWD/istio-1.31.1/bin:$PATH"
istioctl install --set profile=minimal -y
kubectl -n istio-system rollout status deployment/istiod --timeout=180s

Install Vault in dev mode

Bash

helm repo add hashicorp https://helm.releases.hashicorp.com
helm repo update
helm install vault hashicorp/vault --version 0.34.1 -n vault \
  --set server.dev.enabled=true --set injector.enabled=true
kubectl -n vault wait --for=condition=ready pod/vault-0 --timeout=240s
kubectl -n vault get pods

Expected result: vault-0 and the vault-agent-injector pod are running. Dev mode runs Vault as a StatefulSet pod named vault-0, which is why you wait on the pod, not on a Deployment.

A helper for Vault commands

Vault commands run inside the pod. This function saves typing and sets the dev-mode root token.

Bash

vexec() { kubectl exec -n vault vault-0 -- env VAULT_TOKEN=root vault "$@"; }
vexec status

Expected result: Sealed false and a version number.

Common Mistake
Running kubectl rollout status deployment/vault. In dev mode there is no Deployment, so the command fails with "not found". Wait on the pod instead, as above.

Milestone 2: Deploy Postgres and Configure Dynamic Credentials

This milestone deploys a small Postgres database and teaches Vault how to create temporary users in it. Vault needs one admin login to do that; applications never see it.

Deploy Postgres in the data namespace

YAML

# postgres.yaml
apiVersion: v1
kind: Secret
metadata:
  name: postgres-admin
  namespace: data
type: Opaque
stringData:
  password: change-me-lab-only
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: postgres
  namespace: data
spec:
  replicas: 1
  selector:
    matchLabels:
      app: postgres
  template:
    metadata:
      labels:
        app: postgres
    spec:
      containers:
        - name: postgres
          image: postgres:16-alpine
          env:
            - name: POSTGRES_DB
              value: payments
            - name: POSTGRES_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: postgres-admin
                  key: password
          ports:
            - containerPort: 5432
---
apiVersion: v1
kind: Service
metadata:
  name: postgres
  namespace: data
spec:
  selector:
    app: postgres
  ports:
    - port: 5432

Bash

kubectl apply -f postgres.yaml
kubectl -n data rollout status deployment/postgres --timeout=180s
kubectl -n data exec deploy/postgres -- psql -U postgres -d payments -c \
  "CREATE TABLE payments (id serial PRIMARY KEY, amount int); INSERT INTO payments (amount) VALUES (10), (25);"

Enable the database secrets engine

Bash

PGPASS="$(kubectl -n data get secret postgres-admin -o jsonpath='{.data.password}' | base64 -d)"
 
vexec secrets enable database
 
vexec write database/config/payments \
  plugin_name=postgresql-database-plugin \
  allowed_roles=app \
  connection_url='postgresql://{{username}}:{{password}}@postgres.data.svc.cluster.local:5432/payments?sslmode=disable' \
  username=postgres \
  password="$PGPASS"
 
vexec write database/roles/app \
  db_name=payments \
  creation_statements="CREATE ROLE \"{{name}}\" WITH LOGIN PASSWORD '{{password}}' VALID UNTIL '{{expiration}}'; GRANT SELECT ON ALL TABLES IN SCHEMA public TO \"{{name}}\";" \
  default_ttl=2m \
  max_ttl=10m

The role creates read-only users that live for 2 minutes and can be renewed up to 10 minutes. A real service would use longer lifetimes; the short ones let you watch expiry and renewal during this lab.

Test it by hand

Bash

vexec read database/creds/app

Expected result: a lease ID, a lease duration of 2m, and a generated username that starts with v-. That user now exists in Postgres and will be deleted by Vault when the lease ends.

Security
The admin password above is a lab value. In a real setup, rotate it so only Vault knows it, with vault write -f database/rotate-root/payments, and store nothing else that can log in as admin.

Common Mistake
Giving the dynamic role broad rights such as ALL PRIVILEGES. Each generated user gets exactly the statements in creation_statements, so write them as the narrowest grant the service needs.

Milestone 3: Inject Credentials Into a Pod With Kubernetes Auth

This milestone lets a pod prove its identity to Vault using its Kubernetes service account, then receive database credentials in a file. The Vault Agent Injector adds a helper container to the pod that logs in, fetches credentials, renews them, and rewrites the file when they change. The application needs no Vault code.

Configure Kubernetes authentication

This command runs inside the Vault pod, which has its own service account and the cluster address in its environment, so no extra values are needed. The single quotes make the variable resolve inside the pod.

Bash

vexec auth enable kubernetes
 
kubectl exec -n vault vault-0 -- sh -c \
  'VAULT_TOKEN=root vault write auth/kubernetes/config kubernetes_host="https://$KUBERNETES_PORT_443_TCP_ADDR:443"'
 
kubectl exec -i -n vault vault-0 -- env VAULT_TOKEN=root vault policy write db-reader - << 'EOF'
path "database/creds/app" {
  capabilities = ["read"]
}
EOF
 
vexec write auth/kubernetes/role/db-reader \
  bound_service_account_names=db-reader \
  bound_service_account_namespaces=payments \
  policies=db-reader \
  ttl=1h

The role says: only the service account db-reader in the namespace payments may log in, and it may only read database/creds/app.

Deploy a pod that uses the credentials

The container loops forever, reads the connection string from the injected file, and prints which database user it is. The Istio annotation excludes Vault's port from the mesh so the Vault agent can reach Vault before the Istio sidecar is ready.

YAML

# db-reader.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
  name: db-reader
  namespace: payments
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: db-reader
  namespace: payments
spec:
  replicas: 1
  selector:
    matchLabels:
      app: db-reader
  template:
    metadata:
      labels:
        app: db-reader
      annotations:
        vault.hashicorp.com/agent-inject: "true"
        vault.hashicorp.com/role: "db-reader"
        vault.hashicorp.com/agent-inject-secret-db-url: "database/creds/app"
        vault.hashicorp.com/agent-inject-template-db-url: |
          {{- with secret "database/creds/app" -}}
          postgresql://{{ .Data.username }}:{{ .Data.password }}@postgres.data.svc.cluster.local:5432/payments?sslmode=disable
          {{- end }}
        traffic.sidecar.istio.io/excludeOutboundPorts: "8200"
    spec:
      serviceAccountName: db-reader
      containers:
        - name: reader
          image: postgres:16-alpine
          command: ["sh", "-c"]
          args:
            - |
              while true; do
                psql "$(cat /vault/secrets/db-url)" -Atc "select current_user, count(*) from payments" || echo "query failed"
                sleep 15
              done

Bash

kubectl apply -f db-reader.yaml
kubectl -n payments rollout status deployment/db-reader --timeout=240s
kubectl -n payments logs deploy/db-reader -c reader --tail=5

Expected result: lines such as v-...|2, a generated username and the row count. Wait three or four minutes and read the logs again: the username changes when the lease ends and the agent fetches a new one.

Confirm the secret is not in the environment

Bash

kubectl -n payments exec deploy/db-reader -c reader -- env | grep -i -E 'pass|secret|token' || echo "nothing in env"

Expected result: nothing in env. The credential lives in a file under /vault/secrets, which is easier to rotate and keeps it out of process listings. Anyone who can exec into the pod can still read that file, which is why exec access needs to stay tightly controlled.

Common Mistake
Mounting the injected file but never reading it again. The agent rewrites the file when credentials rotate, so the application must reread it for each new connection. An app that reads it once at startup breaks when the first lease ends.

Milestone 4: Enforce mTLS and Identity-Based Access in Istio

This milestone turns on strict mTLS in the payments namespace and adds an authorization policy that lets only one named workload call the API. Identity comes from the service account of each pod, carried inside the mTLS certificate.

Deploy the API and two callers

YAML

# mesh-workloads.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
  name: api
  namespace: payments
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: client
  namespace: payments
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: intruder
  namespace: payments
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: api
  namespace: payments
spec:
  replicas: 1
  selector:
    matchLabels:
      app: api
  template:
    metadata:
      labels:
        app: api
    spec:
      serviceAccountName: api
      containers:
        - name: api
          image: ghcr.io/stefanprodan/podinfo:6.15.0
          ports:
            - containerPort: 9898
---
apiVersion: v1
kind: Service
metadata:
  name: api
  namespace: payments
spec:
  selector:
    app: api
  ports:
    - port: 9898
      targetPort: 9898
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: client
  namespace: payments
spec:
  replicas: 1
  selector:
    matchLabels:
      app: client
  template:
    metadata:
      labels:
        app: client
    spec:
      serviceAccountName: client
      containers:
        - name: curl
          image: curlimages/curl:8.11.1
          command: ["sleep", "86400"]
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: intruder
  namespace: payments
spec:
  replicas: 1
  selector:
    matchLabels:
      app: intruder
  template:
    metadata:
      labels:
        app: intruder
    spec:
      serviceAccountName: intruder
      containers:
        - name: curl
          image: curlimages/curl:8.11.1
          command: ["sleep", "86400"]

Bash

kubectl apply -f mesh-workloads.yaml
kubectl -n payments rollout status deployment/api deployment/client deployment/intruder --timeout=240s
kubectl -n payments get pods

Expected result: the api, client and intruder pods show 2/2 containers ready, the app plus the Istio sidecar. The db-reader pod shows 3/3 because it also carries the Vault agent.

Require mTLS and allow only the client

YAML

# mesh-policy.yaml
apiVersion: security.istio.io/v1
kind: PeerAuthentication
metadata:
  name: default
  namespace: payments
spec:
  mtls:
    mode: STRICT
---
apiVersion: security.istio.io/v1
kind: AuthorizationPolicy
metadata:
  name: api-allow-client
  namespace: payments
spec:
  selector:
    matchLabels:
      app: api
  action: ALLOW
  rules:
    - from:
        - source:
            principals: ["cluster.local/ns/payments/sa/client"]

Bash

kubectl apply -f mesh-policy.yaml

With an ALLOW policy in place for the API, any request that does not match a rule is denied. You do not need a separate deny-all policy for this workload.

Test three callers

Bash

CALL='curl -s -o /dev/null -w "%{http_code}\n" http://api:9898/healthz'
kubectl -n payments exec deploy/client -c curl -- sh -c "$CALL"
kubectl -n payments exec deploy/intruder -c curl -- sh -c "$CALL"
kubectl run probe -n legacy --rm -i --restart=Never --image=curlimages/curl:8.11.1 --command -- \
  curl -sS -m 5 http://api.payments.svc.cluster.local:9898/healthz

Expected result: 200 for the client, 403 for the intruder (authenticated but not allowed), and a connection error for the legacy pod (no sidecar, so no certificate, so refused under STRICT mode).

Caller	Has a certificate?	Allowed by policy?	Result
client	Yes	Yes	200
intruder	Yes	No	403
legacy probe	No	Not evaluated	Connection refused

Common Mistake
Turning on STRICT mode before every pod in the namespace has a sidecar. A pod without one is cut off immediately. Start with PERMISSIVE, confirm every pod shows its sidecar, then switch to STRICT.

Tip
This namespace deliberately has no Pod Security restricted label. Istio's init container needs extra network capabilities, which the restricted standard rejects. To combine them, use the Istio CNI plugin instead of the init container, which moves that privilege to a node component.

Milestone 5: Run Security Chaos Experiments

Security chaos engineering means breaking things on purpose to check that controls hold. Each experiment starts with a hypothesis, which turns random breakage into a test with a clear pass or fail.

The experiments

Experiment	Hypothesis
Kill the db-reader pod	The replacement pod gets a new database user automatically and queries succeed
Revoke every lease	The agent notices and the pod continues with a new user within a few minutes
Delete the policy, then restore it	While the policy is missing the intruder gets through, and after restoring it gets 403 again

Experiment 1: kill the pod

Bash

kubectl -n payments logs deploy/db-reader -c reader --tail=1
kubectl -n payments delete pod -l app=db-reader
kubectl -n payments rollout status deployment/db-reader --timeout=240s
kubectl -n payments logs deploy/db-reader -c reader --tail=3

Expected result: the new pod prints a different v-... username than the old one, and the row count query succeeds.

Experiment 2: revoke all leases

Bash

vexec lease revoke -prefix database/creds/app
sleep 200
kubectl -n payments logs deploy/db-reader -c reader --tail=10

Expected result: at least one query failed line may appear while the old user is gone, followed by successful lines with a new username once the agent fetches fresh credentials. The recovery time depends on the 2 minute lease, so this is a good moment to discuss why production leases and renewal behaviour need real testing.

Experiment 3: remove and restore the policy

Bash

kubectl -n payments delete authorizationpolicy api-allow-client
kubectl -n payments exec deploy/intruder -c curl -- sh -c 'curl -s -o /dev/null -w "%{http_code}\n" http://api:9898/healthz'
kubectl apply -f mesh-policy.yaml
sleep 5
kubectl -n payments exec deploy/intruder -c curl -- sh -c 'curl -s -o /dev/null -w "%{http_code}\n" http://api:9898/healthz'

Expected result: 200 while the policy is missing, then 403 after it is restored. The first answer is the lesson: without an ALLOW policy Istio allows authenticated traffic, so the policy file in version control is the control, and deleting it silently weakens security.

Common Mistake
Running chaos without a written hypothesis. You then cannot tell whether the result was a surprise or the expected behaviour. Write the sentence first, run the experiment, then record whether it held.

Security
Only run these on a cluster you own. Revoking leases and deleting policies on a shared environment can cause a real outage.

Validation & Testing

This script checks every control after the experiments and exits non-zero if any fails. Keep it with your project as evidence that the platform still behaves as designed.

Bash

#!/usr/bin/env bash
# verify.sh
set -u
pass=0; fail=0
ok()  { echo "PASS: $1"; pass=$((pass+1)); }
bad() { echo "FAIL: $1"; fail=$((fail+1)); }
code() { kubectl -n payments exec "deploy/$1" -c curl -- sh -c 'curl -s -o /dev/null -w "%{http_code}" http://api:9898/healthz'; }
 
[ "$(code client)" = "200" ] && ok "client can call api" || bad "client can call api"
[ "$(code intruder)" = "403" ] && ok "intruder gets 403" || bad "intruder gets 403"
 
mode="$(kubectl -n payments get peerauthentication default -o jsonpath='{.spec.mtls.mode}')"
[ "${mode}" = "STRICT" ] && ok "mTLS is STRICT" || bad "mTLS is STRICT"
 
user="$(kubectl -n payments logs deploy/db-reader -c reader --tail=1)"
echo "${user}" | grep -q '^v-' && ok "db-reader uses a Vault-generated user" || bad "db-reader uses a Vault-generated user"
 
kubectl -n payments exec deploy/db-reader -c reader -- env | grep -qi -E 'pass|secret|token' \
  && bad "no secrets in environment" || ok "no secrets in environment"
 
echo "passed=${pass} failed=${fail}"
[ "${fail}" -eq 0 ]

Bash

chmod +x verify.sh && ./verify.sh

Expected result: five PASS lines and failed=0.

Checklist

Vault dev pod, Istio control plane and Postgres are running.
vault read database/creds/app returns a short-lived user and the user disappears after the lease ends.
The db-reader pod logs show the database username changing over time.
Client gets 200, intruder gets 403, and the legacy probe cannot connect.
All three chaos experiments have a recorded hypothesis and result.

Reflection questions

What can an attacker do with a leaked dynamic credential compared with a static one?
What does mTLS prove that an IP allow-list cannot?
Why does deleting an authorization policy open access, and how would you stop that happening by accident?
What would you change before using Vault outside dev mode?

Portfolio deliverable

Your README should contain the architecture diagram, the three-caller test table with real output, the chaos hypotheses and results, and a short "what I would change for production" list covering Vault storage and auto-unseal, TLS to Vault, and rotating the root database credential.

Tip
Stretch goals: add the Istio CNI plugin and Pod Security restricted, rotate the Postgres root credential through Vault, and run the verify script on a schedule.

Found a mistake, an outdated step or a broken link?

Zero-Trust Security with Vault and Istio mTLS

Blueprint Walkthrough

Architecture Overview & Problem Statement

Key ideas

Scope and prerequisites

Milestone 1: Create the Cluster and Install Istio and Vault

Create the cluster and namespaces

Install Istio

Install Vault in dev mode

A helper for Vault commands

Milestone 2: Deploy Postgres and Configure Dynamic Credentials

Deploy Postgres in the data namespace

Enable the database secrets engine

Test it by hand

Milestone 3: Inject Credentials Into a Pod With Kubernetes Auth

Configure Kubernetes authentication

Deploy a pod that uses the credentials

Confirm the secret is not in the environment

Milestone 4: Enforce mTLS and Identity-Based Access in Istio

Deploy the API and two callers

Require mTLS and allow only the client

Test three callers

Milestone 5: Run Security Chaos Experiments

The experiments

Experiment 1: kill the pod

Experiment 2: revoke all leases

Experiment 3: remove and restore the policy

Validation & Testing

Checklist

Reflection questions

Portfolio deliverable