This project solves one of the most critical security problems in production engineering — hardcoded credentials. You will deploy HashiCorp Vault on Kubernetes, configure it to generate dynamic database credentials that automatically expire, and use the Vault Agent Sidecar Injector to deliver those credentials directly into application pods without the application ever knowing Vault exists. This is the exact secrets management architecture used by security-conscious engineering teams at Razorpay, Zerodha, and CRED. When a Zerodha trading service needs to connect to PostgreSQL, it never has a password in its environment variables or config files. Vault generates a unique username and password for that specific pod, valid for exactly 1 hour, and automatically rotates it. Internet / Developers | v +------------------+ | AWS KMS Key | <- Auto-unseals Vault on restart +------------------+ | v +------------------+ | Vault Cluster | <- Runs in vault namespace | (3 replicas) | +------------------+ | | +------+ +--------+ | | v v +------------+ +---------------+ | PostgreSQL | | Vault Agent | | (dynamic | | Sidecar in | | creds) | | app pods | +------------+ +---------------+ | v +----------------+ | Application | | reads secret | | from file | +----------------+
Hardcoded credentials are the number one cause of production security incidents. A developer accidentally commits a database password to GitHub — it gets indexed by bots within seconds, attackers connect to the production database, and a crisis begins. This happens to companies of every size. The traditional fix — environment variables — is only marginally better. Secrets in environment variables are visible to anyone who can run `kubectl describe pod` or `printenv` inside a container. Vault with dynamic secrets eliminates both problems completely: * No password exists until the pod starts — Vault generates it on demand. * The password expires automatically — even if stolen, it becomes useless within the TTL window. * Rotation is automatic — applications never need to be redeployed to get new credentials. * Every credential access is logged — full audit trail of which pod accessed which secret and when.
### Step 1: Set Up the Prerequisites Before installing Vault you need a running Kubernetes cluster and an AWS account for KMS auto-unseal. Vault normally requires manual unsealing after every restart — AWS KMS eliminates this by holding the unseal key automatically. ```bash ## Verify your cluster is running kubectl cluster-info kubectl get nodes ## Install Helm (Vault is deployed via Helm) curl https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3 | bash helm version ## Add the HashiCorp Helm repository helm repo add hashicorp https://helm.releases.hashicorp.com helm repo update ``` **Create the AWS KMS key for auto-unseal:** ```bash ## Create a dedicated KMS key for Vault unseal aws kms create-key \ --description "Vault Auto-Unseal Key - Production" \ --key-usage ENCRYPT_DECRYPT \ --region ap-south-1 ## Note the KeyId from the output — you will need it shortly ## Format: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx ## Create an alias for easier reference aws kms create-alias \ --alias-name alias/vault-unseal \ --target-key-id YOUR_KEY_ID \ --region ap-south-1 ## Create an IAM policy that allows Vault to use this key cat > vault-kms-policy.json << 'EOF' { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "kms:Encrypt", "kms:Decrypt", "kms:DescribeKey" ], "Resource": "arn:aws:kms:ap-south-1:YOUR_ACCOUNT_ID:key/YOUR_KEY_ID" } ] } EOF aws iam create-policy \ --policy-name VaultKMSUnsealPolicy \ --policy-document file://vault-kms-policy.json ``` > 📌 **Remember:** The KMS key must be in the same AWS region as your EKS cluster. Cross-region KMS calls add latency and cost. Keep everything in ap-south-1 for this project. ### Step 2: Install Vault on Kubernetes Create the Vault Helm values file. This configures Vault with 3 replicas for high availability, integrated storage (Raft) so you do not need a separate etcd or Consul, and AWS KMS for automatic unsealing. Create `vault-values.yaml`: ```yaml ## vault-values.yaml global: enabled: true tlsDisable: false # Always enable TLS in production injector: enabled: true # This is the Vault Agent Sidecar Injector replicas: 1 server: replicas: 3 # High availability — 3 Vault instances # Use integrated Raft storage — no external dependency needed ha: enabled: true replicas: 3 raft: enabled: true config: | ui = true listener "tcp" { tls_disable = 1 # TLS terminated at ingress in this setup address = "[::]:8200" cluster_address = "[::]:8201" } storage "raft" { path = "/vault/data" retry_join { leader_api_addr = "http://vault-0.vault-internal:8200" } retry_join { leader_api_addr = "http://vault-1.vault-internal:8200" } retry_join { leader_api_addr = "http://vault-2.vault-internal:8200" } } # AWS KMS Auto-Unseal configuration seal "awskms" { region = "ap-south-1" kms_key_id = "alias/vault-unseal" } service_registration "kubernetes" {} # Service account with AWS IAM role for KMS access serviceAccount: annotations: eks.amazonaws.com/role-arn: "arn:aws:iam::YOUR_ACCOUNT_ID:role/vault-kms-role" # Persistent storage for Raft data dataStorage: enabled: true size: 10Gi storageClass: gp2 ui: enabled: true serviceType: ClusterIP # Expose via ingress, not directly ``` ```bash ## Create the vault namespace kubectl create namespace vault ## Install Vault helm install vault hashicorp/vault \ --namespace vault \ --values vault-values.yaml \ --version 0.27.0 ## Watch pods come up (takes 2-3 minutes) kubectl get pods -n vault --watch ## Expected output after a few minutes: ## vault-0 0/1 Running 0 60s ## vault-1 0/1 Running 0 60s ## vault-2 0/1 Running 0 60s ## vault-agent-injector-xxx 1/1 Running 0 60s ``` ### Step 3: Initialize and Verify Vault Vault starts in a sealed state the first time. You need to initialize it to generate the root token and recovery keys. With AWS KMS auto-unseal configured, subsequent restarts unseal automatically. ```bash ## Initialize Vault on the first pod kubectl exec -n vault vault-0 -- vault operator init \ -key-shares=1 \ -key-threshold=1 ## CRITICAL: Save the output carefully. It contains: ## Recovery Key 1: xxxx (save this securely — needed for disaster recovery) ## Initial Root Token: hvs.xxxx (use this to log in) ## Check Vault status — should show initialized: true, sealed: false ## (KMS auto-unseal means it unseals automatically) kubectl exec -n vault vault-0 -- vault status ## Log into Vault using the root token kubectl exec -n vault vault-0 -- vault login hvs.YOUR_ROOT_TOKEN ## Verify all 3 nodes joined the Raft cluster kubectl exec -n vault vault-0 -- vault operator raft list-peers ## Expected: 3 peers all showing alive ``` > ⚠️ **Security:** The root token has unlimited permissions. In production, create a less-privileged admin token immediately after initialization and revoke the root token. Store recovery keys in separate secure locations — AWS Secrets Manager, a password manager, and offline cold storage. ### Step 4: Enable Kubernetes Authentication Vault needs to verify that a request is coming from a legitimate Kubernetes pod. The Kubernetes auth method does this by validating the pod's ServiceAccount JWT token against the Kubernetes API. ```bash ## Enable Kubernetes authentication in Vault kubectl exec -n vault vault-0 -- vault auth enable kubernetes ## Configure it to talk to the Kubernetes API kubectl exec -n vault vault-0 -- vault write auth/kubernetes/config \ kubernetes_host="https://kubernetes.default.svc.cluster.local:443" ## Verify it was configured correctly kubectl exec -n vault vault-0 -- vault read auth/kubernetes/config ``` **Create a Vault policy that defines what the application is allowed to access:** ```bash ## Create a policy file cat > app-policy.hcl << 'EOF' ## Allow the application to read database credentials path "database/creds/webapp-role" { capabilities = ["read"] } ## Allow the application to renew its own token path "auth/token/renew-self" { capabilities = ["update"] } EOF ## Write the policy to Vault kubectl exec -n vault vault-0 -- vault policy write webapp-policy - < app-policy.hcl ## Create a Kubernetes auth role linking the ServiceAccount to the policy kubectl exec -n vault vault-0 -- vault write auth/kubernetes/role/webapp \ bound_service_account_names=webapp \ bound_service_account_namespaces=default \ policies=webapp-policy \ ttl=1h ``` ### Step 5: Configure Dynamic Database Secrets This is the most powerful part of Vault. Instead of storing a static password, Vault connects to PostgreSQL as an admin and creates unique users with expiring passwords on demand. ```bash ## First, deploy a PostgreSQL instance for testing kubectl apply -f - <<EOF apiVersion: apps/v1 kind: Deployment metadata: name: postgres spec: replicas: 1 selector: matchLabels: app: postgres template: metadata: labels: app: postgres spec: containers: * name: postgres image: postgres:15-alpine env: * name: POSTGRES_PASSWORD value: "vault-admin-password" # Vault uses this admin password internally * name: POSTGRES_DB value: "appdb" ports: * containerPort: 5432 --- apiVersion: v1 kind: Service metadata: name: postgres spec: selector: app: postgres ports: * port: 5432 EOF ## Enable the database secrets engine in Vault kubectl exec -n vault vault-0 -- vault secrets enable database ## Configure Vault to connect to PostgreSQL as an admin kubectl exec -n vault vault-0 -- vault write database/config/postgresql \ plugin_name=postgresql-database-plugin \ allowed_roles="webapp-role" \ connection_url="postgresql://postgres:vault-admin-password@postgres.default.svc.cluster.local:5432/appdb?sslmode=disable" \ username="postgres" \ password="vault-admin-password" ## Create a role — this is the template Vault uses to create dynamic users kubectl exec -n vault vault-0 -- vault write database/roles/webapp-role \ db_name=postgresql \ creation_statements="CREATE ROLE \"{{name}}\" WITH LOGIN PASSWORD '{{password}}' VALID UNTIL '{{expiration}}'; \ GRANT SELECT, INSERT, UPDATE ON ALL TABLES IN SCHEMA public TO \"{{name}}\";" \ default_ttl="1h" \ max_ttl="24h" ## Test it — generate a dynamic credential manually kubectl exec -n vault vault-0 -- vault read database/creds/webapp-role ## Expected output: ## lease_id database/creds/webapp-role/xxxxxxxx ## lease_duration 1h ## username v-kubernetes-webapp-xxxxxxxx (unique generated username) ## password A1B2C3D4-xxxx-xxxx-xxxx (unique generated password) ``` > 💡 **Tip:** Run `vault read database/creds/webapp-role` twice and compare the output. You get a completely different username and password each time. Both are valid simultaneously until they expire. This is dynamic secrets — no static password exists anywhere. ### Step 6: Deploy Application with Vault Agent Sidecar Injection Now deploy an application that automatically receives database credentials without any Vault SDK code. The Vault Agent Sidecar runs as a second container in the pod, authenticates to Vault, retrieves the secret, and writes it to a shared file. The application reads from the file. ```bash ## Create the ServiceAccount the app will use kubectl create serviceaccount webapp ## Deploy the application with Vault annotations kubectl apply -f - <<EOF apiVersion: apps/v1 kind: Deployment metadata: name: webapp spec: replicas: 2 selector: matchLabels: app: webapp template: metadata: labels: app: webapp annotations: # These annotations tell the injector to inject Vault Agent vault.hashicorp.com/agent-inject: "true" vault.hashicorp.com/role: "webapp" # Must match the role created in Vault vault.hashicorp.com/agent-inject-secret-db-creds: "database/creds/webapp-role" vault.hashicorp.com/agent-inject-template-db-creds: | {{- with secret "database/creds/webapp-role" -}} DB_USERNAME={{ .Data.username }} DB_PASSWORD={{ .Data.password }} DB_HOST=postgres.default.svc.cluster.local DB_PORT=5432 DB_NAME=appdb {{- end }} spec: serviceAccountName: webapp # Must match the ServiceAccount in the Vault role containers: * name: webapp image: nginx:alpine # Replace with your actual application image # The application reads credentials from this file: # /vault/secrets/db-creds # Format: KEY=VALUE pairs (like a .env file) EOF ## Watch the pods start — you should see 2 containers per pod kubectl get pods ## Expected: webapp-xxx 2/2 Running ## The 2/2 means: 1 app container + 1 Vault Agent sidecar ## Verify the credentials were injected correctly kubectl exec -it webapp-POD_NAME -c webapp -- cat /vault/secrets/db-creds ## Expected output: ## DB_USERNAME=v-kubernetes-webapp-xxxxxxxx ## DB_PASSWORD=A1B2C3-xxxx ## DB_HOST=postgres.default.svc.cluster.local ## DB_PORT=5432 ## DB_NAME=appdb ``` > 🔴 **Common Mistake:** The ServiceAccount name in the Kubernetes deployment (`serviceAccountName: webapp`) must exactly match the `bound_service_account_names` configured in the Vault Kubernetes role. If they do not match, the Vault Agent cannot authenticate and the pod will be stuck in Init state. ### Step 7: Verify Audit Logging One of Vault's most powerful features is complete audit logging of every secret access. Enable it and verify it captures credential generation. ```bash ## Enable file-based audit logging kubectl exec -n vault vault-0 -- vault audit enable file \ file_path=/vault/logs/audit.log ## Generate a new database credential to create a log entry kubectl exec -n vault vault-0 -- vault read database/creds/webapp-role ## Read the audit log kubectl exec -n vault vault-0 -- cat /vault/logs/audit.log | head -50 ## You will see a JSON entry showing: ## - Which token made the request ## - Which path was accessed (database/creds/webapp-role) ## - The timestamp ## - The client IP ## This is your complete credential access audit trail ```
```bash ## 1. Verify Vault cluster health kubectl exec -n vault vault-0 -- vault status ## Expected: Initialized=true, Sealed=false, HA Enabled=true ## 2. Verify Raft cluster has 3 healthy nodes kubectl exec -n vault vault-0 -- vault operator raft list-peers ## Expected: 3 peers, all with state voter and leader/follower ## 3. Generate dynamic credentials and verify they work CREDS=$(kubectl exec -n vault vault-0 -- vault read -format=json database/creds/webapp-role) USER=$(echo $CREDS | python3 -c "import sys,json; print(json.load(sys.stdin)['data']['username'])") PASS=$(echo $CREDS | python3 -c "import sys,json; print(json.load(sys.stdin)['data']['password'])") echo "Generated: $USER" ## 4. Verify the credential expires (run after 1 hour or reduce TTL for testing) ## Set a short TTL for testing: kubectl exec -n vault vault-0 -- vault write database/roles/webapp-role \ db_name=postgresql \ creation_statements="CREATE ROLE \"{{name}}\" WITH LOGIN PASSWORD '{{password}}' VALID UNTIL '{{expiration}}'; GRANT SELECT ON ALL TABLES IN SCHEMA public TO \"{{name}}\";" \ default_ttl="2m" \ max_ttl="5m" ## Generate credential, wait 2 minutes, verify it no longer works in PostgreSQL ## 5. Test KMS auto-unseal by restarting a Vault pod kubectl delete pod vault-0 -n vault ## Watch it restart and automatically unseal (should take 30-60 seconds) kubectl get pod vault-0 -n vault --watch kubectl exec -n vault vault-0 -- vault status ## Expected: Sealed=false (auto-unsealed by KMS without any manual intervention) ## 6. Verify webapp pod has credentials injected kubectl exec -it deployment/webapp -c webapp -- cat /vault/secrets/db-creds ## Expected: valid DB_USERNAME, DB_PASSWORD, DB_HOST entries ## 7. Verify no secrets exist in pod environment variables kubectl exec -it deployment/webapp -c webapp -- env | grep -i password ## Expected: NO OUTPUT — no passwords in environment variables echo "Zero hardcoded credentials confirmed" ```
This project solves one of the most critical security problems in production engineering — hardcoded credentials. You wi...
Hardcoded credentials are the number one cause of production security incidents. A developer accidentally commits a data...
Step 1: Set Up the Prerequisites Before installing Vault you need a running Kubernetes cluster and an AWS account for KM...
...
Aligns directly with DevOps, Site Reliability (SRE), and Platform Engineering job descriptions.