Build a Production Secrets Management System with HashiCorp Vault and Kubernetes

Deploy HashiCorp Vault on Kubernetes, configure dynamic secrets, Vault Agent injection, and AWS KMS auto-unseal for zero hardcoded credentials.

Related Concepts & TermsDeployment Environment Variable grep IAM Role KMS Key Helm

Domains & Technologies

Domains

VAULTSECURITY

Technologies

KUBERNETESAWS

Blueprint Walkthrough

Architecture Overview

This project solves one of the most critical security problems in production engineering — hardcoded credentials. You will deploy HashiCorp Vault on Kubernetes, configure it to generate dynamic database credentials that automatically expire, and use the Vault Agent Sidecar Injector to deliver those credentials directly into application pods without the application ever knowing Vault exists.

This is the exact secrets management architecture used by security-conscious engineering teams at Razorpay, Zerodha, and CRED. When a Zerodha trading service needs to connect to PostgreSQL, it never has a password in its environment variables or config files. Vault generates a unique username and password for that specific pod, valid for exactly 1 hour, and automatically rotates it.

◈ DIAGRAM

    Internet / Developers
             |
             v
      +------------------+
      |   AWS KMS Key    |  <- Auto-unseals Vault on restart
      +------------------+
             |
             v
      +------------------+
      |  Vault Cluster   |  <- Runs in vault namespace
      |  (3 replicas)    |
      +------------------+
             |        |
      +------+        +--------+
      |                        |
      v                        v
+------------+         +---------------+
| PostgreSQL |         | Vault Agent   |
| (dynamic   |         | Sidecar in    |
| creds)     |         | app pods      |
+------------+         +---------------+
                              |
                              v
                     +----------------+
                     | Application    |
                     | reads secret   |
                     | from file      |
                     +----------------+

Problem Solved

Hardcoded credentials are the number one cause of production security incidents. A developer accidentally commits a database password to GitHub — it gets indexed by bots within seconds, attackers connect to the production database, and a crisis begins. This happens to companies of every size.

The traditional fix — environment variables — is only marginally better. Secrets in environment variables are visible to anyone who can run kubectl describe pod or printenv inside a container.

Vault with dynamic secrets eliminates both problems completely:

No password exists until the pod starts — Vault generates it on demand.
The password expires automatically — even if stolen, it becomes useless within the TTL window.
Rotation is automatic — applications never need to be redeployed to get new credentials.
Every credential access is logged — full audit trail of which pod accessed which secret and when.

Step-by-Step Implementation Guide

Step 1: Set Up the Prerequisites

Before installing Vault you need a running Kubernetes cluster and an AWS account for KMS auto-unseal. Vault normally requires manual unsealing after every restart — AWS KMS eliminates this by holding the unseal key automatically.

Bash

## Verify your cluster is running
kubectl cluster-info
kubectl get nodes
 
## Install Helm (Vault is deployed via Helm)
curl https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3 | bash
helm version
 
## Add the HashiCorp Helm repository
helm repo add hashicorp https://helm.releases.hashicorp.com
helm repo update

Create the AWS KMS key for auto-unseal:

Bash

## Create a dedicated KMS key for Vault unseal
aws kms create-key \
  --description "Vault Auto-Unseal Key - Production" \
  --key-usage ENCRYPT_DECRYPT \
  --region ap-south-1
 
## Note the KeyId from the output — you will need it shortly
## Format: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
 
## Create an alias for easier reference
aws kms create-alias \
  --alias-name alias/vault-unseal \
  --target-key-id YOUR_KEY_ID \
  --region ap-south-1
 
## Create an IAM policy that allows Vault to use this key
cat > vault-kms-policy.json << 'EOF'
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "kms:Encrypt",
        "kms:Decrypt",
        "kms:DescribeKey"
      ],
      "Resource": "arn:aws:kms:ap-south-1:YOUR_ACCOUNT_ID:key/YOUR_KEY_ID"
    }
  ]
}
EOF
 
aws iam create-policy \
  --policy-name VaultKMSUnsealPolicy \
  --policy-document file://vault-kms-policy.json

Remember
The KMS key must be in the same AWS region as your EKS cluster. Cross-region KMS calls add latency and cost. Keep everything in ap-south-1 for this project.

Step 2: Install Vault on Kubernetes

Create the Vault Helm values file. This configures Vault with 3 replicas for high availability, integrated storage (Raft) so you do not need a separate etcd or Consul, and AWS KMS for automatic unsealing.

Create vault-values.yaml:

YAML

## vault-values.yaml
global:
  enabled: true
  tlsDisable: false  # Always enable TLS in production
 
injector:
  enabled: true  # This is the Vault Agent Sidecar Injector
  replicas: 1
 
server:
  replicas: 3  # High availability — 3 Vault instances
 
  # Use integrated Raft storage — no external dependency needed
  ha:
    enabled: true
    replicas: 3
    raft:
      enabled: true
      config: |
        ui = true
        listener "tcp" {
          tls_disable = 1  # TLS terminated at ingress in this setup
          address = "[::]:8200"
          cluster_address = "[::]:8201"
        }
 
        storage "raft" {
          path = "/vault/data"
 
          retry_join {
            leader_api_addr = "http://vault-0.vault-internal:8200"
          }
          retry_join {
            leader_api_addr = "http://vault-1.vault-internal:8200"
          }
          retry_join {
            leader_api_addr = "http://vault-2.vault-internal:8200"
          }
        }
 
        # AWS KMS Auto-Unseal configuration
        seal "awskms" {
          region     = "ap-south-1"
          kms_key_id = "alias/vault-unseal"
        }
 
        service_registration "kubernetes" {}
 
  # Service account with AWS IAM role for KMS access
  serviceAccount:
    annotations:
      eks.amazonaws.com/role-arn: "arn:aws:iam::YOUR_ACCOUNT_ID:role/vault-kms-role"
 
  # Persistent storage for Raft data
  dataStorage:
    enabled: true
    size: 10Gi
    storageClass: gp2
 
ui:
  enabled: true
  serviceType: ClusterIP  # Expose via ingress, not directly

Bash

## Create the vault namespace
kubectl create namespace vault
 
## Install Vault
helm install vault hashicorp/vault \
  --namespace vault \
  --values vault-values.yaml \
  --version 0.27.0
 
## Watch pods come up (takes 2-3 minutes)
kubectl get pods -n vault --watch
 
## Expected output after a few minutes:
## vault-0   0/1   Running   0   60s
## vault-1   0/1   Running   0   60s  
## vault-2   0/1   Running   0   60s
## vault-agent-injector-xxx   1/1   Running   0   60s

Step 3: Initialize and Verify Vault

Vault starts in a sealed state the first time. You need to initialize it to generate the root token and recovery keys. With AWS KMS auto-unseal configured, subsequent restarts unseal automatically.

Bash

## Initialize Vault on the first pod
kubectl exec -n vault vault-0 -- vault operator init \
  -key-shares=1 \
  -key-threshold=1
 
## CRITICAL: Save the output carefully. It contains:
## Recovery Key 1: xxxx (save this securely — needed for disaster recovery)
## Initial Root Token: hvs.xxxx (use this to log in)
 
## Check Vault status — should show initialized: true, sealed: false
## (KMS auto-unseal means it unseals automatically)
kubectl exec -n vault vault-0 -- vault status
 
## Log into Vault using the root token
kubectl exec -n vault vault-0 -- vault login hvs.YOUR_ROOT_TOKEN
 
## Verify all 3 nodes joined the Raft cluster
kubectl exec -n vault vault-0 -- vault operator raft list-peers
## Expected: 3 peers all showing alive

Security
The root token has unlimited permissions. In production, create a less-privileged admin token immediately after initialization and revoke the root token. Store recovery keys in separate secure locations — AWS Secrets Manager, a password manager, and offline cold storage.

Step 4: Enable Kubernetes Authentication

Vault needs to verify that a request is coming from a legitimate Kubernetes pod. The Kubernetes auth method does this by validating the pod's ServiceAccount JWT token against the Kubernetes API.

Bash

## Enable Kubernetes authentication in Vault
kubectl exec -n vault vault-0 -- vault auth enable kubernetes
 
## Configure it to talk to the Kubernetes API
kubectl exec -n vault vault-0 -- vault write auth/kubernetes/config \
  kubernetes_host="https://kubernetes.default.svc.cluster.local:443"
 
## Verify it was configured correctly
kubectl exec -n vault vault-0 -- vault read auth/kubernetes/config

Create a Vault policy that defines what the application is allowed to access:

Bash

## Create a policy file
cat > app-policy.hcl << 'EOF'
## Allow the application to read database credentials
path "database/creds/webapp-role" {
  capabilities = ["read"]
}
 
## Allow the application to renew its own token
path "auth/token/renew-self" {
  capabilities = ["update"]
}
EOF
 
## Write the policy to Vault
kubectl exec -n vault vault-0 -- vault policy write webapp-policy - < app-policy.hcl
 
## Create a Kubernetes auth role linking the ServiceAccount to the policy
kubectl exec -n vault vault-0 -- vault write auth/kubernetes/role/webapp \
  bound_service_account_names=webapp \
  bound_service_account_namespaces=default \
  policies=webapp-policy \
  ttl=1h

Step 5: Configure Dynamic Database Secrets

This is the most powerful part of Vault. Instead of storing a static password, Vault connects to PostgreSQL as an admin and creates unique users with expiring passwords on demand.

Bash

## First, deploy a PostgreSQL instance for testing
kubectl apply -f - <<EOF
apiVersion: apps/v1
kind: Deployment
metadata:
  name: postgres
spec:
  replicas: 1
  selector:
    matchLabels:
      app: postgres
  template:
    metadata:
      labels:
        app: postgres
    spec:
      containers:
      * name: postgres
        image: postgres:15-alpine
        env:
        * name: POSTGRES_PASSWORD
          value: "vault-admin-password"  # Vault uses this admin password internally
        * name: POSTGRES_DB
          value: "appdb"
        ports:
        * containerPort: 5432
---
apiVersion: v1
kind: Service
metadata:
  name: postgres
spec:
  selector:
    app: postgres
  ports:
  * port: 5432
EOF
 
## Enable the database secrets engine in Vault
kubectl exec -n vault vault-0 -- vault secrets enable database
 
## Configure Vault to connect to PostgreSQL as an admin
kubectl exec -n vault vault-0 -- vault write database/config/postgresql \
  plugin_name=postgresql-database-plugin \
  allowed_roles="webapp-role" \
  connection_url="postgresql://postgres:vault-admin-password@postgres.default.svc.cluster.local:5432/appdb?sslmode=disable" \
  username="postgres" \
  password="vault-admin-password"
 
## Create a role — this is the template Vault uses to create dynamic users
kubectl exec -n vault vault-0 -- vault write database/roles/webapp-role \
  db_name=postgresql \
  creation_statements="CREATE ROLE \"{{name}}\" WITH LOGIN PASSWORD '{{password}}' VALID UNTIL '{{expiration}}'; \
                       GRANT SELECT, INSERT, UPDATE ON ALL TABLES IN SCHEMA public TO \"{{name}}\";" \
  default_ttl="1h" \
  max_ttl="24h"
 
## Test it — generate a dynamic credential manually
kubectl exec -n vault vault-0 -- vault read database/creds/webapp-role
## Expected output:
## lease_id          database/creds/webapp-role/xxxxxxxx
## lease_duration    1h
## username          v-kubernetes-webapp-xxxxxxxx  (unique generated username)
## password          A1B2C3D4-xxxx-xxxx-xxxx (unique generated password)

Tip
Run vault read database/creds/webapp-role twice and compare the output. You get a completely different username and password each time. Both are valid simultaneously until they expire. This is dynamic secrets — no static password exists anywhere.

Step 6: Deploy Application with Vault Agent Sidecar Injection

Now deploy an application that automatically receives database credentials without any Vault SDK code. The Vault Agent Sidecar runs as a second container in the pod, authenticates to Vault, retrieves the secret, and writes it to a shared file. The application reads from the file.

Bash

## Create the ServiceAccount the app will use
kubectl create serviceaccount webapp
 
## Deploy the application with Vault annotations
kubectl apply -f - <<EOF
apiVersion: apps/v1
kind: Deployment
metadata:
  name: webapp
spec:
  replicas: 2
  selector:
    matchLabels:
      app: webapp
  template:
    metadata:
      labels:
        app: webapp
      annotations:
        # These annotations tell the injector to inject Vault Agent
        vault.hashicorp.com/agent-inject: "true"
        vault.hashicorp.com/role: "webapp"  # Must match the role created in Vault
        vault.hashicorp.com/agent-inject-secret-db-creds: "database/creds/webapp-role"
        vault.hashicorp.com/agent-inject-template-db-creds: |
          {{- with secret "database/creds/webapp-role" -}}
          DB_USERNAME={{ .Data.username }}
          DB_PASSWORD={{ .Data.password }}
          DB_HOST=postgres.default.svc.cluster.local
          DB_PORT=5432
          DB_NAME=appdb
          {{- end }}
    spec:
      serviceAccountName: webapp  # Must match the ServiceAccount in the Vault role
      containers:
      * name: webapp
        image: nginx:alpine  # Replace with your actual application image
        # The application reads credentials from this file:
        # /vault/secrets/db-creds
        # Format: KEY=VALUE pairs (like a .env file)
EOF
 
## Watch the pods start — you should see 2 containers per pod
kubectl get pods
## Expected: webapp-xxx   2/2   Running
## The 2/2 means: 1 app container + 1 Vault Agent sidecar
 
## Verify the credentials were injected correctly
kubectl exec -it webapp-POD_NAME -c webapp -- cat /vault/secrets/db-creds
## Expected output:
## DB_USERNAME=v-kubernetes-webapp-xxxxxxxx
## DB_PASSWORD=A1B2C3-xxxx
## DB_HOST=postgres.default.svc.cluster.local
## DB_PORT=5432
## DB_NAME=appdb

Common Mistake
The ServiceAccount name in the Kubernetes deployment (serviceAccountName: webapp) must exactly match the bound_service_account_names configured in the Vault Kubernetes role. If they do not match, the Vault Agent cannot authenticate and the pod will be stuck in Init state.

Step 7: Verify Audit Logging

One of Vault's most powerful features is complete audit logging of every secret access. Enable it and verify it captures credential generation.

Bash

## Enable file-based audit logging
kubectl exec -n vault vault-0 -- vault audit enable file \
  file_path=/vault/logs/audit.log
 
## Generate a new database credential to create a log entry
kubectl exec -n vault vault-0 -- vault read database/creds/webapp-role
 
## Read the audit log
kubectl exec -n vault vault-0 -- cat /vault/logs/audit.log | head -50
 
## You will see a JSON entry showing:
## - Which token made the request
## - Which path was accessed (database/creds/webapp-role)
## - The timestamp
## - The client IP
## This is your complete credential access audit trail

Validation & Testing

Bash

## 1. Verify Vault cluster health
kubectl exec -n vault vault-0 -- vault status
## Expected: Initialized=true, Sealed=false, HA Enabled=true
 
## 2. Verify Raft cluster has 3 healthy nodes
kubectl exec -n vault vault-0 -- vault operator raft list-peers
## Expected: 3 peers, all with state voter and leader/follower
 
## 3. Generate dynamic credentials and verify they work
CREDS=$(kubectl exec -n vault vault-0 -- vault read -format=json database/creds/webapp-role)
USER=$(echo $CREDS | python3 -c "import sys,json; print(json.load(sys.stdin)['data']['username'])")
PASS=$(echo $CREDS | python3 -c "import sys,json; print(json.load(sys.stdin)['data']['password'])")
echo "Generated: $USER"
 
## 4. Verify the credential expires (run after 1 hour or reduce TTL for testing)
## Set a short TTL for testing:
kubectl exec -n vault vault-0 -- vault write database/roles/webapp-role \
  db_name=postgresql \
  creation_statements="CREATE ROLE \"{{name}}\" WITH LOGIN PASSWORD '{{password}}' VALID UNTIL '{{expiration}}'; GRANT SELECT ON ALL TABLES IN SCHEMA public TO \"{{name}}\";" \
  default_ttl="2m" \
  max_ttl="5m"
## Generate credential, wait 2 minutes, verify it no longer works in PostgreSQL
 
## 5. Test KMS auto-unseal by restarting a Vault pod
kubectl delete pod vault-0 -n vault
## Watch it restart and automatically unseal (should take 30-60 seconds)
kubectl get pod vault-0 -n vault --watch
kubectl exec -n vault vault-0 -- vault status
## Expected: Sealed=false (auto-unsealed by KMS without any manual intervention)
 
## 6. Verify webapp pod has credentials injected
kubectl exec -it deployment/webapp -c webapp -- cat /vault/secrets/db-creds
## Expected: valid DB_USERNAME, DB_PASSWORD, DB_HOST entries
 
## 7. Verify no secrets exist in pod environment variables
kubectl exec -it deployment/webapp -c webapp -- env | grep -i password
## Expected: NO OUTPUT — no passwords in environment variables
echo "Zero hardcoded credentials confirmed"

Videos & Guides

Video Guide

HashiCorp Vault on Kubernetes — Complete Tutorial

End-to-end tutorial covering Vault installation on Kubernetes, Kubernetes auth method, dynamic secrets configuration, and Vault Agent sidecar injection for production workloads.

View Reference

DOCUMENTATION

Vault Kubernetes Sidecar Injector Documentation

Official HashiCorp documentation for the Vault Agent Injector — covering annotations, templates, init containers, and advanced injection patterns.

View Reference

DOCUMENTATION

Vault Dynamic Secrets — PostgreSQL Engine

Complete reference for configuring the Vault PostgreSQL database secrets engine including role creation, TTL configuration, and credential lifecycle management.

View Reference