Vault Kubernetes Secrets with Dynamic Credentials

Deploy HashiCorp Vault on Kubernetes with Raft HA and KMS auto-unseal, then inject short-lived PostgreSQL credentials into pods with Vault Agent.

Architecture Overview & Problem Statement

This project deploys a three-node HashiCorp Vault cluster on Kubernetes with Raft storage and AWS KMS auto-unseal, then configures Vault to create a unique, short-lived PostgreSQL user for every pod that asks. The Vault Agent injector delivers those credentials to the application as a file, so the application image holds no database password and the Git repository holds none either. You finish by proving that credentials expire, can be revoked on demand, and are recorded in an audit log.

You need an EKS cluster with an OIDC provider (as in Deploy a Production EKS Cluster with Terraform), kubectl, Helm, eksctl, and the AWS CLI.

What you build:

A KMS key and an IAM role so Vault can unseal itself after a restart
A three-node Vault cluster with integrated Raft storage
Kubernetes authentication, so a pod proves who it is with its service account
A database secrets engine that issues expiring PostgreSQL users
An application that receives credentials from a sidecar and never talks to Vault itself

flowchart TD KMS([AWS KMS
auto-unseal]) -->|unseals| V subgraph VNS["Namespace: vault"] V[Vault
3 Raft nodes] INJ[Agent injector] end subgraph ANS["Namespace: default"] POD[App pod
app and Agent sidecar] end INJ -.->|adds sidecar| POD POD -->|login and read| V V -->|creates user| DB[(PostgreSQL)] POD -->|connects| DB classDef cloud fill:#F97316,color:#FFFFFF classDef vault fill:#8B5CF6,color:#FFFFFF classDef app fill:#326CE5,color:#FFFFFF classDef data fill:#10B981,color:#FFFFFF class KMS cloud class V,INJ vault class POD app class DB data

The problem this solves

A database password stored in a Git repository, a container image, or an environment variable can be found by anyone who can read those things. Environment variables are visible through kubectl describe pod and printenv. A leaked static password stays valid until someone notices and rotates it.

With dynamic secrets, no password exists until a workload asks for one. Vault creates a database user on demand, gives it a lease (a time limit), and deletes it when the lease ends. A stolen credential is useful only until the lease expires, and every issue is logged. You will measure these properties yourself in the validation step, so you can state exactly what your setup guarantees.

Tip
"Zero hardcoded credentials" is a goal to verify, not a slogan. The validation step gives you concrete checks: nothing in the pod environment, nothing in Git, and credentials that stop working.

Milestone 1: Create the KMS Key and IAM Role for Auto-Unseal

This milestone creates the key that lets Vault unseal itself. Vault encrypts its data with a master key, and starts sealed, unable to read anything until it is unsealed. By default a human supplies key shares after every restart. With auto-unseal, Vault asks AWS KMS to decrypt its master key, so a restarted pod recovers on its own.

Bash

export CLUSTER=your-cluster-name
export REGION=ap-south-1
export ACCOUNT_ID=$(aws sts get-caller-identity --query Account --output text)
 
export KMS_KEY_ID=$(aws kms create-key \
  --description "Vault auto-unseal" \
  --region $REGION \
  --query KeyMetadata.KeyId --output text)
 
export KMS_KEY_ARN=$(aws kms describe-key --key-id $KMS_KEY_ID --region $REGION \
  --query KeyMetadata.Arn --output text)
 
cat > vault-kms-policy.json <<EOF
{
  "Version": "2012-10-17",
  "Statement": [{
    "Effect": "Allow",
    "Action": ["kms:Encrypt", "kms:Decrypt", "kms:DescribeKey"],
    "Resource": "$KMS_KEY_ARN"
  }]
}
EOF
 
aws iam create-policy \
  --policy-name VaultKMSUnsealPolicy \
  --policy-document file://vault-kms-policy.json

Create the namespace and a service account that carries an IAM role (IRSA). eksctl creates the role, its trust policy, and the annotated service account in one step:

Bash

kubectl create namespace vault
 
eksctl create iamserviceaccount \
  --cluster $CLUSTER \
  --namespace vault \
  --name vault \
  --attach-policy-arn arn:aws:iam::$ACCOUNT_ID:policy/VaultKMSUnsealPolicy \
  --approve

Common Mistake
Creating the IAM policy but never creating the role that Vault assumes. The Helm values then point at a role that does not exist, and Vault fails to start with a KMS permission error. The service account above is what connects the two.

Milestone 2: Install a Three-Node Vault Cluster

This milestone installs Vault in high-availability mode using the official Helm chart. Raft is Vault's built-in storage, so you do not need a separate Consul or etcd. One node is the active leader and two are standbys that take over if it fails.

Look up the current chart version and pin it:

Bash

helm repo add hashicorp https://helm.releases.hashicorp.com
helm repo update
helm search repo hashicorp/vault
export VAULT_CHART_VERSION=<version from the search above>

Create vault-values.yaml. The service account already exists, so the chart must not create its own.

YAML

global:
  enabled: true
  tlsDisable: true   # lab only, see the security note below
 
injector:
  enabled: true
 
server:
  serviceAccount:
    create: false
    name: vault
 
  ha:
    enabled: true
    replicas: 3
    raft:
      enabled: true
      setNodeId: true
      config: |
        ui = true
        listener "tcp" {
          tls_disable     = 1
          address         = "[::]:8200"
          cluster_address = "[::]:8201"
        }
 
        storage "raft" {
          path = "/vault/data"
 
          retry_join {
            leader_api_addr = "http://vault-0.vault-internal:8200"
          }
          retry_join {
            leader_api_addr = "http://vault-1.vault-internal:8200"
          }
          retry_join {
            leader_api_addr = "http://vault-2.vault-internal:8200"
          }
        }
 
        seal "awskms" {
          region     = "ap-south-1"
          kms_key_id = "YOUR_KMS_KEY_ID"
        }
 
        service_registration "kubernetes" {}
 
  dataStorage:
    enabled: true
    size: 10Gi
 
ui:
  enabled: true
  serviceType: ClusterIP

Replace YOUR_KMS_KEY_ID with the value of $KMS_KEY_ID, then install:

Bash

helm install vault hashicorp/vault \
  --namespace vault \
  --values vault-values.yaml \
  --version $VAULT_CHART_VERSION
 
kubectl get pods -n vault --watch
# Expected: vault-0, vault-1, vault-2 Running but 0/1 Ready, plus the injector 1/1

The Vault pods stay 0/1 ready until the cluster is initialized, which is correct.

Security
This lab turns TLS off inside the cluster to keep the focus on secrets. That is not acceptable for production, where Vault traffic must use TLS and the listener, probes, and client addresses all change to https. Treat TLS as the first item on your production list.

Milestone 3: Initialize Vault and Secure the Root Token

This milestone initializes the cluster, which generates the root token and recovery keys. With auto-unseal, KMS holds the unseal key, so what you receive are recovery keys, used only for emergency operations such as generating a new root token. Pass the recovery flags, not the key-share ones.

Bash

kubectl exec -n vault vault-0 -- vault operator init \
  -recovery-shares=1 \
  -recovery-threshold=1 \
  -format=json > vault-init.json

vault-init.json holds the root token and a recovery key. Move it to a password manager or secrets store, then delete it from your disk. Anyone with the root token can do anything in Vault.

The other two nodes join through retry_join and unseal themselves with KMS:

Bash

kubectl get pods -n vault
# Expected: vault-0, vault-1 and vault-2 all 1/1 Ready
 
kubectl exec -n vault vault-0 -- vault status
# Expected: Initialized true, Sealed false, HA Enabled true

Open a shell in the pod for the Vault commands that follow:

Bash

kubectl exec -it -n vault vault-0 -- sh

Inside that shell, log in with the root token, confirm the cluster, and replace the root token with a short-lived admin token:

Bash

# inside the Vault pod
vault login            # paste the root token when asked
 
vault operator raft list-peers
# Expected: three peers, one leader and two followers
 
vault policy write admin - <<'EOF'
path "*" {
  capabilities = ["create", "read", "update", "delete", "list", "sudo"]
}
EOF
 
vault token create -policy=admin -ttl=8h

Log in again with the new admin token, then revoke the root token (vault token revoke ROOT_TOKEN). You can generate a new one later with the recovery key if you ever need it.

Security
An admin policy with * is root in everything but name. It is acceptable for a lab. In production, humans sign in through OIDC or SSO, and policies are narrow.

Milestone 4: Enable Kubernetes Authentication

This milestone lets a pod prove its identity to Vault. A pod presents its service account token, and Vault checks it against the Kubernetes API. If the token belongs to a service account that a Vault role trusts, Vault returns a token carrying only that role's policies.

Run these in the Vault pod shell:

Bash

# inside the Vault pod
vault auth enable kubernetes
 
vault write auth/kubernetes/config \
  kubernetes_host="https://$KUBERNETES_PORT_443_TCP_ADDR:443"

Write a policy that allows reading one credential path and nothing else:

Bash

# inside the Vault pod
vault policy write webapp-policy - <<'EOF'
path "database/creds/webapp-role" {
  capabilities = ["read"]
}
EOF
 
vault write auth/kubernetes/role/webapp \
  bound_service_account_names=webapp \
  bound_service_account_namespaces=default \
  policies=webapp-policy \
  ttl=1h

The role binds one service account name and one namespace. A pod with a different service account cannot obtain these credentials, even if it runs next to this one.

Common Mistake
A mismatch between serviceAccountName in the Deployment and bound_service_account_names in the Vault role. The sidecar cannot log in, and the pod hangs in Init or its sidecar logs a permission denied error. The two names must match exactly.

Milestone 5: Configure Dynamic PostgreSQL Credentials

This milestone connects Vault to a database and defines how it creates users. Vault logs in as an admin, runs your SQL to create a role with an expiry, and drops it when the lease ends.

First deploy a test database. Its admin password lives in a Secret, not in the manifest:

Bash

kubectl create secret generic postgres-root --from-literal=password='temporary-admin-password'
 
kubectl apply -f - <<'EOF'
apiVersion: apps/v1
kind: Deployment
metadata:
  name: postgres
spec:
  replicas: 1
  selector:
    matchLabels:
      app: postgres
  template:
    metadata:
      labels:
        app: postgres
    spec:
      containers:
        - name: postgres
          image: postgres:16-alpine
          env:
            - name: POSTGRES_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: postgres-root
                  key: password
            - name: POSTGRES_DB
              value: appdb
          ports:
            - containerPort: 5432
---
apiVersion: v1
kind: Service
metadata:
  name: postgres
spec:
  selector:
    app: postgres
  ports:
    - port: 5432
EOF
 
kubectl rollout status deployment/postgres
kubectl exec deploy/postgres -- psql -U postgres -d appdb \
  -c "CREATE TABLE orders (id serial PRIMARY KEY, item text);"

Back in the Vault pod shell, enable the engine and register the database. The {{username}} and {{password}} placeholders in the URL are filled in by Vault:

Bash

# inside the Vault pod
vault secrets enable database
 
vault write database/config/postgresql \
  plugin_name=postgresql-database-plugin \
  allowed_roles="webapp-role" \
  connection_url="postgresql://{{username}}:{{password}}@postgres.default.svc.cluster.local:5432/appdb?sslmode=disable" \
  username="postgres" \
  password="temporary-admin-password"
 
# Rotate the admin password so only Vault knows it from now on
vault write -f database/rotate-root/postgresql

Define the role. The SQL goes in a file so the quoting stays readable:

Bash

# inside the Vault pod
cat > /tmp/creation.sql <<'EOF'
CREATE ROLE "{{name}}" WITH LOGIN PASSWORD '{{password}}' VALID UNTIL '{{expiration}}';
GRANT SELECT, INSERT, UPDATE ON ALL TABLES IN SCHEMA public TO "{{name}}";
EOF
 
vault write database/roles/webapp-role \
  db_name=postgresql \
  creation_statements=@/tmp/creation.sql \
  default_ttl=1h \
  max_ttl=24h
 
vault read database/creds/webapp-role
vault read database/creds/webapp-role

Run the last command twice. Each call returns a different username and password, both valid at the same time, each with its own lease.

Tip
rotate-root replaces the admin password you typed with one only Vault knows. Afterwards, even the person who ran this lab cannot log in to the database as that admin.

Milestone 6: Inject Credentials into an Application

This milestone delivers credentials to a pod without any Vault code in the application. The Vault Agent injector is a mutating webhook: when it sees the annotations below, it adds a sidecar that logs in to Vault, fetches the secret, renders it to a file, and renews the lease for as long as it is allowed.

Bash

kubectl create serviceaccount webapp
 
kubectl apply -f - <<'EOF'
apiVersion: apps/v1
kind: Deployment
metadata:
  name: webapp
spec:
  replicas: 2
  selector:
    matchLabels:
      app: webapp
  template:
    metadata:
      labels:
        app: webapp
      annotations:
        vault.hashicorp.com/agent-inject: "true"
        vault.hashicorp.com/role: "webapp"
        vault.hashicorp.com/agent-inject-secret-db-creds: "database/creds/webapp-role"
        vault.hashicorp.com/agent-inject-template-db-creds: |
          {{- with secret "database/creds/webapp-role" -}}
          DB_USERNAME={{ .Data.username }}
          DB_PASSWORD={{ .Data.password }}
          DB_HOST=postgres.default.svc.cluster.local
          DB_PORT=5432
          DB_NAME=appdb
          {{- end }}
    spec:
      serviceAccountName: webapp
      containers:
        - name: webapp
          image: nginx:alpine   # replace with your application image
EOF
 
kubectl get pods -l app=webapp
# Expected: 2/2 Running (the app plus the Vault Agent sidecar)
 
kubectl exec deploy/webapp -c webapp -- cat /vault/secrets/db-creds

The application reads /vault/secrets/db-creds. Credentials rotate as leases renew, so a real application must re-read the file or be signalled when it changes. The injector supports a command annotation for that, and many teams run a short reload hook.

Now prove the credentials work against the database. The connection goes through the service name, which forces password authentication:

Bash

kubectl exec deploy/webapp -c webapp -- cat /vault/secrets/db-creds > /tmp/creds.env
source /tmp/creds.env
 
kubectl exec deploy/postgres -- env PGPASSWORD="$DB_PASSWORD" \
  psql -h postgres.default.svc.cluster.local -U "$DB_USERNAME" -d appdb -c "SELECT 1;"
# Expected: a row with the value 1

Common Mistake
Testing from inside the database pod over 127.0.0.1 or a Unix socket. The default image trusts local connections, so the password is never checked and the test proves nothing. Use the service address, as above.

Milestone 7: Turn On Audit Logging

This milestone records every request to Vault. Audit logging answers "which workload read which secret, and when", which is a core reason to use Vault at all. Send the log to standard output so your normal log collection can pick it up:

Bash

# inside the Vault pod
vault audit enable file file_path=stdout

Bash

kubectl exec -n vault vault-0 -- vault read database/creds/webapp-role
kubectl logs -n vault vault-0 | grep "database/creds/webapp-role" | head -3

Each entry names the path, the time, the client, and the identity that made the request. Secret values are not written in clear text: Vault stores salted hashes, so the log can be shared without leaking credentials.

Validation & Testing

Check each property, and record what you saw.

Bash

# 1. Cluster health
kubectl exec -n vault vault-0 -- vault status
kubectl exec -n vault vault-0 -- vault operator raft list-peers
# Expected: unsealed, HA enabled, three peers
 
# 2. Credentials are unique per request
kubectl exec -n vault vault-0 -- vault read -format=json database/creds/webapp-role | grep username
kubectl exec -n vault vault-0 -- vault read -format=json database/creds/webapp-role | grep username
# Expected: two different usernames
 
# 3. No password in the pod environment
kubectl exec deploy/webapp -c webapp -- env | grep -i password
# Expected: no output
 
# 4. No credential in Git history of your manifests
git log -p | grep -i "temporary-admin-password"
# Expected: no output (the admin password was rotated and lives only in Vault)

Expiry. Shorten the lifetime, issue a credential, and confirm it stops working:

Bash

# inside the Vault pod
vault write database/roles/webapp-role db_name=postgresql \
  creation_statements=@/tmp/creation.sql default_ttl=2m max_ttl=5m
vault read database/creds/webapp-role

Test the new login with the psql command from Milestone 6, wait about five minutes, and repeat it. Expected: the second attempt fails.

Revocation. Revoke every credential at once, as you would after a suspected leak:

Bash

# inside the Vault pod
vault lease revoke -prefix database/creds/webapp-role

Bash

kubectl exec deploy/postgres -- psql -U postgres -d appdb -c "\du"
# Expected: the generated v-... roles are gone

Auto-unseal. Delete a pod and time the recovery:

Bash

kubectl delete pod vault-0 -n vault
kubectl get pod vault-0 -n vault --watch
kubectl exec -n vault vault-0 -- vault status
# Expected: Sealed false, with no human action

Write your results in a table in the README:

Property	Target	Measured
Credential lifetime in the app	Matches `default_ttl`
Time for a restarted node to unseal	Under a few minutes
Credentials usable after expiry	None
Credentials usable after revoke	None
Passwords found in env, image, or Git	None

In your portfolio, include the architecture diagram, the audit log excerpt, and the completed table. Add a short list of what production needs beyond this lab: TLS on every Vault listener, narrow human policies behind SSO, network policies around the Vault namespace, scheduled Raft snapshots (vault operator raft snapshot save), and an alert if the audit device fails.

Clean up with helm uninstall vault -n vault, delete the namespace, and schedule the KMS key for deletion. Remember that the Raft volumes hold encrypted data until you delete the persistent volume claims.

Found a mistake, an outdated step or a broken link?

Vault Kubernetes Secrets with Dynamic Credentials

Blueprint Walkthrough

Architecture Overview & Problem Statement

The problem this solves

Milestone 1: Create the KMS Key and IAM Role for Auto-Unseal

Milestone 2: Install a Three-Node Vault Cluster

Milestone 3: Initialize Vault and Secure the Root Token

Milestone 4: Enable Kubernetes Authentication

Milestone 5: Configure Dynamic PostgreSQL Credentials

Milestone 6: Inject Credentials into an Application

Milestone 7: Turn On Audit Logging

Validation & Testing