Managing Kubernetes Secrets with Vault and ConfigMaps

Managing Kubernetes Secrets with Vault and ConfigMaps

Overview and What You Will Learn

Hardcoding credentials inside Docker images or environment variables is one of the most dangerous and common security mistakes in Kubernetes deployments. This lab walks you through the correct production approach — managing sensitive credentials using Kubernetes Secrets with encryption at rest, injecting secrets dynamically using HashiCorp Vault, and separating non-sensitive configuration using ConfigMaps with proper RBAC controls.

By the end of this guide you will be able to:

• Create and consume Kubernetes Secrets safely as environment variables and mounted files
• Enable encryption at rest for Secrets stored in etcd
• Deploy HashiCorp Vault on Kubernetes and inject secrets dynamically into pods
• Use ConfigMaps for non-sensitive application configuration with automatic reload
• Apply RBAC policies to restrict which pods and service accounts can access which secrets

Why This Matters in Production

In 2023, a major Indian fintech platform suffered a credential leak because database passwords were stored as plain environment variables in their Kubernetes Deployment YAML files — which were committed to a public GitHub repository. The blast radius included customer PII exposure and regulatory penalties.

At Zerodha, Razorpay, and PhonePe — platforms handling real financial transactions — secrets management is a compliance requirement, not just a best practice. A single leaked database credential or API key can compromise millions of user accounts. Every engineer deploying to Kubernetes must understand how to handle credentials correctly from day one.

Core Principles

The secrets management hierarchy in a production Kubernetes cluster:

Key rules every engineer must follow:

• Never store sensitive values in ConfigMaps — they are plain text in etcd
• Never commit Secret YAML files to Git — use Sealed Secrets or Vault instead
• Always enable encryption at rest for the Secrets API group in etcd
• Always use RBAC to restrict which service accounts can read which secrets
• Rotate secrets regularly — Vault enables automatic rotation without pod restarts

Detailed Step-by-Step Practical Lab

Step 1 — Create and Use Basic Kubernetes Secrets

Step 2 — Inject Secrets into Pods as Environment Variables

Step 3 — Mount Secrets as Files for Certificate and Key Management

⚠️ Security: Always set defaultMode: 0400 on secret volume mounts. The default mode 0644 makes private keys readable by all users on the container — a significant security risk.

Step 4 — Enable Encryption at Rest for Secrets in etcd

By default, Kubernetes Secrets are stored as base64 in etcd — not encrypted. Anyone with etcd access can read all secrets. Fix this:

📌 Remember: Encryption at rest only protects secrets from direct etcd database access. It does not protect secrets from being read via the Kubernetes API by authorised users. RBAC controls who can read secrets via the API.

Step 5 — Deploy HashiCorp Vault on Kubernetes

⚠️ Security: Never store vault-init-keys.json on the same cluster or any version control system. Store unseal keys in separate secure locations — ideally split across different team members using Shamir's Secret Sharing.

Step 6 — Configure Vault Agent Injector for Automatic Secret Injection

Step 7 — Apply RBAC to Restrict Secret Access

💡 Tip: Always use resourceNames in your RBAC Role to restrict access to specific named secrets. Without it, the service account can read ALL secrets in the namespace — including other teams' credentials.

Production Best Practices & Common Pitfalls

• Use External Secrets Operator as an alternative to Vault for teams already using AWS Secrets Manager or GCP Secret Manager — it syncs external secrets into Kubernetes Secrets automatically.
• Never use kubectl get secret -o yaml on a production terminal that is being screen-recorded or shared. The base64 values are trivially decodable.
• Rotate all secrets after any team member leaves the organisation. Vault makes this seamless with dynamic secrets that auto-expire.
• Use Sealed Secrets (by Bitnami) if you must commit secret manifests to Git — it encrypts secrets with a cluster-specific key so the YAML is safe to store in version control.
• Audit secret access regularly using Kubernetes audit logs — any unexpected get secret event from an unrecognised service account is a red flag.

🔴 Common Mistake: Using envFrom: secretRef to inject an entire secret as environment variables. This injects ALL keys including ones the application does not need, violating the principle of least privilege. Always inject only the specific keys your application requires using secretKeyRef.

Quick Reference & Troubleshooting Commands