DevSecOps project 3 — The Platform Security Challenge

The mega-capstone. Wire Terraform IaC scanning, secrets management with Vault, zero-trust mTLS, security chaos engineering, and a unified compliance dashboard into one production security platform.

Domains & Technologies

Domains

CAPSTONEDEVSECOPSVAULTISTIOMTLSCHAOS-ENGINEERING

Technologies

TERRAFORM

Blueprint Walkthrough

The Question That Exposes Gaps

A CISO at a fintech startup walked into the engineering team's standup and asked one question: "If I randomly terminate 20% of our containers right now, will any secrets get exposed?"

Nobody could answer. They had secrets in Kubernetes Secrets (base64-encoded, not encrypted). They had TLS at the ingress but plaintext traffic between internal services. They had security tools installed but no centralized view of their security posture. They had infrastructure as code but no scanning to ensure it stayed secure.

Each of those gaps is a separate problem. This capstone treats them as one problem: building a cohesive security platform where every piece reinforces the others.

You are building five components that integrate into a unified platform:

◈ DIAGRAM

┌────────────────────────────────────────────────────────────┐
│ Component 1: Terraform Security (tfsec + Checkov)          │
│ Scan IaC before provisioning — catch misconfigs at source  │
├────────────────────────────────────────────────────────────┤
│ Component 2: HashiCorp Vault                               │
│ Dynamic secrets — no static credentials anywhere           │
├────────────────────────────────────────────────────────────┤
│ Component 3: Istio mTLS + Zero-Trust Networking            │
│ Every service-to-service call encrypted and authenticated  │
├────────────────────────────────────────────────────────────┤
│ Component 4: Security Chaos Engineering (Chaos Monkey)     │
│ Deliberately break things to test security controls        │
├────────────────────────────────────────────────────────────┤
│ Component 5: Unified Compliance Dashboard                  │
│ Single pane of glass across all security tools             │
└────────────────────────────────────────────────────────────┘

Before You Start

What this capstone builds on

This capstone is the integration layer. It assumes:

Capstone 1: You have a Secure CI/CD pipeline (the gated pipeline from Capstone 1)
Capstone 2: You have a hardened Kubernetes cluster (RBAC, NetworkPolicies, Falco from Capstone 2)

The new components in this capstone connect into both of those. Vault provides the secrets that the CI/CD pipeline and Kubernetes workloads use. Istio sits on top of the hardened cluster. The compliance dashboard aggregates findings from Trivy, Semgrep, Falco, kube-bench, and the new IaC scanners.

Bash

## Verify the prerequisites are running
kubectl cluster-info
kubectl get pods -n falco | grep Running
kubectl get networkpolicies -n payment-service
 
## Install additional tools for this capstone
## Terraform
wget -O /tmp/terraform.zip \
  https://releases.hashicorp.com/terraform/1.7.0/terraform_1.7.0_linux_amd64.zip
unzip /tmp/terraform.zip -d /usr/local/bin/
 
## tfsec — Terraform security scanner
wget -q -O /usr/local/bin/tfsec \
  https://github.com/aquasecurity/tfsec/releases/download/v1.28.4/tfsec-linux-amd64
chmod +x /usr/local/bin/tfsec
 
## Checkov — multi-framework IaC scanner
pip3 install checkov --break-system-packages
 
## Verify all tools
terraform version && tfsec --version && checkov --version
 
echo "✅ Prerequisites verified"

Component 1 — Infrastructure as Code Security

Why IaC misconfigurations are expensive to fix later

Infrastructure as Code is both a security strength and a security risk. The strength: every infrastructure change is in version control, reviewable, auditable. The risk: one misconfigured Terraform module can provision hundreds of insecure resources across all environments simultaneously.

An S3 bucket with acl = "public-read" in a Terraform module gets deployed to dev, staging, and production in the same terraform apply. If the misconfiguration is not caught in the CI pipeline, it sits there until someone notices a billing anomaly or — worse — a breach.

tfsec and Checkov scan Terraform code before it is applied. They catch misconfigurations that are expensive to fix after provisioning: S3 buckets without server-side encryption, security groups open to 0.0.0.0/0, EKS clusters without private API endpoints, RDS instances without deletion protection.

Writing the infrastructure code

Bash

## Set up the IaC project structure
mkdir terraform-infrastructure && cd terraform-infrastructure
mkdir -p modules/eks modules/rds modules/vpc modules/s3 environments/production
touch main.tf variables.tf outputs.tf
touch modules/eks/main.tf modules/rds/main.tf
 
echo "✅ IaC project structure created"

HCL

## modules/eks/main.tf
## EKS cluster configuration
## This file intentionally contains misconfigurations that the scanner catches
 
terraform {
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 5.0"
    }
  }
}
 
variable "cluster_name" {
  type        = string
  description = "Name of the EKS cluster"
}
 
variable "cluster_version" {
  type        = string
  description = "Kubernetes version for the cluster"
  default     = "1.29"
}
 
## MISCONFIGURATION 1: Public API endpoint enabled
## tfsec will flag: aws-eks-no-public-cluster-access
## The Kubernetes API server should not be publicly accessible
resource "aws_eks_cluster" "main" {
  name     = var.cluster_name
  version  = var.cluster_version
  role_arn = aws_iam_role.eks_cluster.arn
 
  vpc_config {
    subnet_ids = var.subnet_ids
 
    ## ❌ Public endpoint should be false in production
    endpoint_public_access  = true
    endpoint_private_access = true
 
    ## ❌ API access is open to all IPs — should be restricted to VPN CIDR
    public_access_cidrs = ["0.0.0.0/0"]
  }
 
  ## MISCONFIGURATION 2: Secrets not encrypted with KMS
  ## tfsec will flag: aws-eks-enable-control-plane-logging
  ## encryption_config block is missing entirely
}
 
## MISCONFIGURATION 3: CloudWatch logging not enabled
## tfsec will flag: aws-eks-enable-control-plane-logging
resource "aws_eks_cluster" "logging_disabled" {
  name = "${var.cluster_name}-bad"
 
  enabled_cluster_log_types = []   ## ❌ should include ["api", "audit", "authenticator"]
}

HCL

## modules/rds/main.tf
## RDS database configuration with deliberate misconfigurations
 
variable "db_identifier" {
  type = string
}
 
## MISCONFIGURATION 4: RDS instance publicly accessible
## tfsec will flag: aws-rds-no-public-db-access
resource "aws_db_instance" "payment_db" {
  identifier = var.db_identifier
  engine     = "postgres"
  engine_version = "15.4"
 
  ## ❌ Production databases should never be publicly accessible
  publicly_accessible    = true
 
  ## ❌ Deletion protection prevents accidental deletion in production
  deletion_protection    = false
 
  ## ❌ Automated backups provide point-in-time recovery
  backup_retention_period = 0
 
  storage_encrypted = true   ## This one is correct ✅
}

Running the IaC security scans

Bash

## ── tfsec: deep Terraform security analysis ──────────────────
tfsec modules/ --format=json --out=tfsec-results.json
 
## View high-severity findings only
tfsec modules/ --minimum-severity HIGH
 
## Expected output:
##
## Result #1 HIGH
## ──────────────────────────────────────────────────────────────
## ID              aws-eks-no-public-cluster-access
## Impact          EKS Kubernetes API server accessible to any IP
## Resolution      Restrict public access CIDRs to specific ranges
## More Info       https://aquasecurity.github.io/tfsec/...
##
## modules/eks/main.tf:22-34
## 22 resource "aws_eks_cluster" "main" {
## ...
## 30       public_access_cidrs = ["0.0.0.0/0"]
##
## Passed: 3  Failed: 4  Ignored: 0

Bash

## ── Checkov: multi-framework IaC scanning ─────────────────────
## Checkov checks Terraform, Kubernetes YAML, Dockerfile, CloudFormation
checkov -d . \
  --framework terraform \
  --output json \
  --output-file-path checkov-results.json \
  --soft-fail  ## run without blocking — generate report first
 
## Count failures by severity
python3 -c "
import json
with open('checkov-results.json') as f:
    results = json.load(f)
 
checks = results.get('results', {})
failed = checks.get('failed_checks', [])
passed = checks.get('passed_checks', [])
 
severity_count = {}
for check in failed:
    sev = check.get('severity', 'UNKNOWN')
    severity_count[sev] = severity_count.get(sev, 0) + 1
 
print(f'Total passed: {len(passed)}')
print(f'Total failed: {len(failed)}')
print()
print('Failed by severity:')
for sev, count in sorted(severity_count.items()):
    print(f'  {sev}: {count}')
"

Fixing the IaC misconfigurations

HCL

## modules/eks/main.tf — SECURE VERSION
 
resource "aws_eks_cluster" "main" {
  name     = var.cluster_name
  version  = var.cluster_version
  role_arn = aws_iam_role.eks_cluster.arn
 
  ## ✅ Enable all audit logging
  enabled_cluster_log_types = ["api", "audit", "authenticator", "controllerManager", "scheduler"]
 
  vpc_config {
    subnet_ids = var.subnet_ids
 
    ## ✅ Private only — API server not exposed to internet
    endpoint_public_access  = false
    endpoint_private_access = true
  }
 
  ## ✅ Encrypt Kubernetes secrets in etcd with KMS
  encryption_config {
    provider {
      key_arn = aws_kms_key.eks_secrets.arn
    }
    resources = ["secrets"]
  }
}
 
## ✅ Dedicated KMS key for EKS secret encryption
resource "aws_kms_key" "eks_secrets" {
  description             = "KMS key for EKS secrets encryption"
  deletion_window_in_days = 7
  enable_key_rotation     = true   ## rotate the key annually
 
  tags = {
    Name        = "${var.cluster_name}-eks-secrets"
    Environment = "production"
    ManagedBy   = "terraform"
  }
}

HCL

## modules/rds/main.tf — SECURE VERSION
 
resource "aws_db_instance" "payment_db" {
  identifier = var.db_identifier
  engine     = "postgres"
  engine_version = "15.4"
 
  ## ✅ Never publicly accessible
  publicly_accessible = false
 
  ## ✅ Prevent accidental deletion — disable only for deliberate teardown
  deletion_protection = true
 
  ## ✅ 7 days of automated backups for point-in-time recovery
  backup_retention_period = 7
 
  ## ✅ Encrypt data at rest
  storage_encrypted = true
  kms_key_id        = aws_kms_key.rds.arn
 
  ## ✅ Enable enhanced monitoring and Performance Insights
  monitoring_interval = 60
  performance_insights_enabled = true
}

Bash

## Add tfsec to the CI/CD pipeline from Capstone 1
## Add this as the first job in secure-pipeline.yml
## (after secrets-detection, before sast-scan)
cat >> .github/workflows/secure-pipeline.yml << 'EOF'
 
  ## ── IaC Security Scan (add after secrets-detection) ──────────
  iac-scan:
    name: "Gate 0.5 — IaC Security Scan (tfsec + Checkov)"
    runs-on: ubuntu-latest
    needs: secrets-detection
    steps:
      - name: Checkout code
        uses: actions/checkout@v4
 
      - name: Run tfsec
        uses: aquasecurity/tfsec-action@v1.0.3
        with:
          working_directory: terraform-infrastructure/
          severity: HIGH
          soft_fail: false
 
      - name: Run Checkov
        uses: bridgecrewio/checkov-action@master
        with:
          directory: terraform-infrastructure/
          framework: terraform
          soft_fail: false
          output_format: sarif
          output_file_path: checkov-results.sarif
EOF
 
echo "✅ IaC scanning added to pipeline"

Component 2 — HashiCorp Vault for Dynamic Secrets

Why static secrets are the root cause of most breaches

Every rotation schedule for static secrets has the same failure mode: the rotation does not happen on schedule because it breaks something, and "temporary" static secrets end up permanent for months or years.

HashiCorp Vault eliminates static secrets by making credentials dynamic. Instead of your application knowing the database password, Vault generates a unique database credential for each application instance, valid for 1 hour. When the lease expires, Vault automatically revokes it. A leaked credential is useless within an hour.

Bash

## Install Vault in development mode for this capstone
## Production Vault uses HA configuration with cloud storage
 
helm repo add hashicorp https://helm.releases.hashicorp.com
 
## Install Vault in dev mode (single node, in-memory storage)
## Replace with production HA config for real deployments
helm install vault hashicorp/vault \
  --namespace vault \
  --create-namespace \
  --set "server.dev.enabled=true" \
  --set "injector.enabled=true"
 
## Wait for Vault to be ready
kubectl rollout status deployment/vault -n vault --timeout=120s
 
## Port-forward for initial configuration
kubectl port-forward svc/vault 8200:8200 -n vault &
export VAULT_ADDR="http://localhost:8200"
export VAULT_TOKEN="root"   ## dev mode uses "root" as the token
 
echo "✅ Vault installed"

Configuring Vault for the payment service

Bash

## Enable the PostgreSQL database secrets engine
vault secrets enable database
 
## Configure the PostgreSQL connection
## Vault connects to the database with admin credentials
## and creates/revokes credentials on behalf of applications
vault write database/config/payment-db \
  plugin_name=postgresql-database-plugin \
  allowed_roles="payment-service" \
  connection_url="postgresql://{{username}}:{{password}}@postgres.database.svc:5432/payments" \
  username="vault_admin" \
  password="$(kubectl get secret postgres-admin -n database -o jsonpath='{.data.password}' | base64 -d)"
 
## Create a role that defines what credentials Vault generates
vault write database/roles/payment-service \
  db_name=payment-db \
  creation_statements="
    CREATE ROLE \"{{name}}\" WITH LOGIN PASSWORD '{{password}}' VALID UNTIL '{{expiration}}';
    GRANT SELECT, INSERT, UPDATE ON payments TO \"{{name}}\";
    GRANT SELECT ON merchants TO \"{{name}}\";
  " \
  revocation_statements="
    REVOKE ALL ON ALL TABLES IN SCHEMA public FROM \"{{name}}\";
    DROP ROLE IF EXISTS \"{{name}}\";
  " \
  ## Credentials valid for 1 hour — Vault renews automatically until max_ttl
  default_ttl="1h" \
  max_ttl="24h"
 
## Test: generate a credential manually
vault read database/creds/payment-service
## Output:
## Key                Value
## ---                -----
## lease_id           database/creds/payment-service/KX...
## lease_duration     1h
## username           v-payment-KoiE8L8e
## password           A1B-random-generated-credential
 
echo "✅ Database secrets engine configured"

Kubernetes auth and the Vault Agent Injector

Vault Agent Injector uses a Kubernetes mutating webhook to inject a sidecar container into pods. The sidecar authenticates to Vault using the pod's ServiceAccount token and writes secrets to a shared volume. The application reads secrets from files — no code changes needed.

Bash

## Enable Kubernetes authentication in Vault
vault auth enable kubernetes
 
## Configure Kubernetes auth with the cluster's details
vault write auth/kubernetes/config \
  kubernetes_host="https://$(kubectl get svc kubernetes -o jsonpath='{.spec.clusterIP}'):443" \
  kubernetes_ca_cert=@<(kubectl get secret \
    $(kubectl get sa vault -n vault -o jsonpath='{.secrets[0].name}') \
    -n vault -o jsonpath='{.data.ca\.crt}' | base64 -d) \
  token_reviewer_jwt="$(kubectl create token vault -n vault)"
 
## Create a Vault policy for the payment service
## This policy says: the payment service can read its own database credentials
vault policy write payment-service - << 'EOF'
path "database/creds/payment-service" {
  capabilities = ["read"]
}
 
path "kv/data/payment-service/*" {
  capabilities = ["read"]
}
EOF
 
## Bind the Kubernetes ServiceAccount to the Vault policy
vault write auth/kubernetes/role/payment-service \
  bound_service_account_names=payment-service \
  bound_service_account_namespaces=payment-service \
  policies=payment-service \
  ttl=1h
 
echo "✅ Vault Kubernetes auth configured"

YAML

## deployments/payment-service-with-vault.yaml
## The Vault Agent Injector annotations tell the webhook to inject
## a sidecar that fetches credentials from Vault
 
apiVersion: apps/v1
kind: Deployment
metadata:
  name: payment-service
  namespace: payment-service
spec:
  replicas: 2
  selector:
    matchLabels:
      app: payment-service
  template:
    metadata:
      labels:
        app: payment-service
      annotations:
        ## These annotations trigger the Vault Agent Injector webhook
        vault.hashicorp.com/agent-inject: "true"
        vault.hashicorp.com/role: "payment-service"
 
        ## Inject the database credentials as a file at /vault/secrets/db-creds
        vault.hashicorp.com/agent-inject-secret-db-creds: "database/creds/payment-service"
        ## Template the credentials into a format the app can source
        vault.hashicorp.com/agent-inject-template-db-creds: |
          {{- with secret "database/creds/payment-service" -}}
          export DB_USERNAME="{{ .Data.username }}"
          export DB_PASSWORD="{{ .Data.password }}"
          {{- end }}
    spec:
      serviceAccountName: payment-service
      containers:
      - name: payment-service
        image: ghcr.io/razorpay/payment-service:abc1234
        command: ["/bin/sh", "-c"]
        args:
          ## Source the Vault-injected credentials before starting the app
          - |
            source /vault/secrets/db-creds
            python src/app.py
        env:
        - name: DB_HOST
          value: "postgres.database.svc.cluster.local"
        - name: DB_PORT
          value: "5432"
        ## DB_USERNAME and DB_PASSWORD come from Vault via /vault/secrets/db-creds
        securityContext:
          allowPrivilegeEscalation: false
          readOnlyRootFilesystem: true
          runAsNonRoot: true
          runAsUser: 10001
          capabilities:
            drop: ["ALL"]
        resources:
          requests:
            cpu: "100m"
            memory: "128Mi"
          limits:
            cpu: "500m"
            memory: "256Mi"
        volumeMounts:
        - name: tmp-dir
          mountPath: /tmp
      volumes:
      - name: tmp-dir
        emptyDir: {}

Bash

## Verify Vault injection is working
kubectl apply -f deployments/payment-service-with-vault.yaml
kubectl rollout status deployment/payment-service -n payment-service
 
## Check that credentials were injected
kubectl exec -n payment-service \
  $(kubectl get pods -n payment-service -l app=payment-service -o name | head -1) \
  -c payment-service \
  -- cat /vault/secrets/db-creds
## Expected output:
## export DB_USERNAME="v-payment-KoiE8L8e"
## export DB_PASSWORD="A1B-xYz-random-2024"
 
echo "✅ Vault dynamic secrets working"

Component 3 — Istio mTLS and Zero-Trust Networking

The problem with trusting internal traffic

Zero-trust security means "never trust, always verify" — including traffic from inside your own network. In a traditional Kubernetes setup without service mesh, if pod A can reach pod B on port 8080, there is no authentication happening at the transport layer. Pod B trusts the traffic because it comes from inside the cluster.

mTLS (mutual TLS) fixes this: both sides of every connection present certificates and verify each other's identity. Pod A proves it is allowed to talk to pod B. Pod B proves it is the real payment service, not an impersonating pod.

Istio is a service mesh that implements mTLS transparently using sidecar proxies (Envoy). Applications do not change. The Envoy sidecar handles all certificate management, rotation, and verification automatically.

Bash

## Install Istio
curl -L https://istio.io/downloadIstio | \
  ISTIO_VERSION=1.20.1 TARGET_ARCH=x86_64 sh -
 
export PATH="$PWD/istio-1.20.1/bin:$PATH"
 
## Install Istio in the cluster with default profile
## Default = pilot (control plane) + ingress gateway
istioctl install --set profile=default -y
 
## Verify Istio is running
kubectl get pods -n istio-system
## Expected: istiod and istio-ingressgateway in Running state
 
echo "✅ Istio installed"

Bash

## Enable Istio sidecar injection for the payment-service namespace
## This label tells Istio to automatically inject Envoy sidecar into all new pods
kubectl label namespace payment-service istio-injection=enabled
 
## Restart pods to get sidecars injected
kubectl rollout restart deployment/payment-service -n payment-service
 
## Verify sidecar was injected — each pod should have 2 containers now
kubectl get pods -n payment-service
## Expected: READY 2/2 (app container + envoy sidecar)

YAML

## istio/peer-authentication.yaml
## PeerAuthentication enforces mTLS for all traffic within the namespace
## STRICT mode means plaintext connections are rejected — mTLS only
 
apiVersion: security.istio.io/v1beta1
kind: PeerAuthentication
metadata:
  name: default
  namespace: payment-service
spec:
  ## mtls.mode: STRICT = reject all non-mTLS connections
  ## This means even other pods without Istio sidecars cannot connect
  mtls:
    mode: STRICT

YAML

## istio/authorization-policy.yaml
## AuthorizationPolicy controls which services can communicate
## This is layer 7 (HTTP) access control on top of mTLS
 
apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
  name: payment-service-policy
  namespace: payment-service
spec:
  selector:
    matchLabels:
      app: payment-service
  action: ALLOW
  rules:
    ## Only allow traffic from the ingress gateway (external customers)
    - from:
      - source:
          ## The principal is the mTLS identity of the caller
          ## This format: cluster.local/ns/NAMESPACE/sa/SERVICEACCOUNT
          principals:
            - "cluster.local/ns/istio-system/sa/istio-ingressgateway-service-account"
      to:
        - operation:
            methods: ["GET", "POST"]
            paths: ["/payment/*", "/health"]

YAML

## istio/deny-all-default.yaml
## Start with deny-all and add explicit allows
## Any traffic not matching an ALLOW policy is denied
 
apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
  name: deny-all
  namespace: payment-service
spec:
  ## No rules = deny all traffic
  {}

Bash

## Apply all Istio policies
kubectl apply -f istio/
 
## Test that mTLS is enforced
## A pod without a sidecar should be rejected
 
kubectl run no-sidecar-test \
  --image=curlimages/curl:8.4.0 \
  --restart=Never \
  --namespace=default \
  -- curl -s --connect-timeout 3 \
     http://payment-service.payment-service.svc.cluster.local:8080/health
 
kubectl logs no-sidecar-test
## Expected: connection refused or RBAC denied (mTLS strict mode rejects plaintext)
 
## Verify mTLS is active from within the mesh
istioctl x describe pod \
  $(kubectl get pods -n payment-service -l app=payment-service -o name | head -1) \
  -n payment-service | grep -i mtls
## Expected: mTLS is STRICT
 
echo "✅ Istio mTLS zero-trust networking active"

Component 4 — Security Chaos Engineering

Testing your controls by breaking things

Security chaos engineering is the practice of deliberately injecting security failures into your systems to verify that controls work and alerts fire. Regular chaos experiments answer the question: "Does our security posture actually hold up under realistic conditions, or do we just think it does?"

The approach uses LitmusChaos — a Kubernetes-native chaos engineering platform — to inject failures, and then verifies that:

Falco detected the suspicious activity
The application recovered cleanly
No secrets were exposed during the chaos
mTLS prevented unauthorized connections during the disruption

Bash

## Install LitmusChaos
kubectl apply -f https://litmuschaos.github.io/litmus/litmus-operator-v3.5.0.yaml
 
## Wait for LitmusChaos to be ready
kubectl rollout status deployment/chaos-operator-ce \
  -n litmus --timeout=120s
 
## Install chaos experiments library
kubectl apply -f \
  https://hub.litmuschaos.io/api/chaos/3.5.0?file=charts/generic/experiments.yaml \
  -n payment-service
 
echo "✅ LitmusChaos installed"

YAML

## chaos/pod-kill-experiment.yaml
## Chaos Experiment 1: Kill random pods in the payment namespace
## Validates: pod recovery time, no secret exposure during restart,
##            Vault reinjects credentials to new pods successfully
 
apiVersion: litmuschaos.io/v1alpha1
kind: ChaosEngine
metadata:
  name: payment-pod-kill
  namespace: payment-service
spec:
  appinfo:
    appns: payment-service
    applabel: "app=payment-service"
    appkind: deployment
  chaosServiceAccount: litmus-admin
  experiments:
  - name: pod-delete
    spec:
      components:
        env:
          ## Kill 1 pod every 10 seconds for 60 seconds total
          - name: TOTAL_CHAOS_DURATION
            value: "60"
          - name: CHAOS_INTERVAL
            value: "10"
          ## Kill 50% of the matching pods at each interval
          - name: FORCE
            value: "false"

YAML

## chaos/network-loss-experiment.yaml
## Chaos Experiment 2: Introduce 50% packet loss to the payment service
## Validates: service resilience, error rates stay below SLA,
##            Falco does not fire false positives on slow connections
 
apiVersion: litmuschaos.io/v1alpha1
kind: ChaosEngine
metadata:
  name: payment-network-loss
  namespace: payment-service
spec:
  appinfo:
    appns: payment-service
    applabel: "app=payment-service"
    appkind: deployment
  chaosServiceAccount: litmus-admin
  experiments:
  - name: pod-network-loss
    spec:
      components:
        env:
          - name: TOTAL_CHAOS_DURATION
            value: "60"
          ## 50% packet loss — service should still respond (slowly)
          - name: NETWORK_PACKET_LOSS_PERCENTAGE
            value: "50"
          ## Target only egress traffic to the database
          - name: DESTINATION_IPS
            value: "$(kubectl get svc postgresql -n database -o jsonpath='{.spec.clusterIP}')"

Bash

## Run the security chaos experiment
kubectl apply -f chaos/pod-kill-experiment.yaml
 
## Monitor the experiment status
kubectl get chaosresult payment-pod-kill-pod-delete \
  -n payment-service -o jsonpath='{.status.verdict}'
 
## While chaos runs — verify these in parallel:
 
## 1. Check Vault is reinjecting credentials to new pods
kubectl logs -n vault -l app.kubernetes.io/name=vault-agent-injector \
  --since=2m | grep "renewed\|injected"
 
## 2. Check no secrets are in environment variables (they should be in files)
kubectl exec -n payment-service \
  $(kubectl get pods -n payment-service -l app=payment-service -o name | head -1) \
  -- env | grep -i "password\|secret\|token\|key"
## Expected: NO matches — secrets are in /vault/secrets/, not env vars
 
## 3. Check Falco did not alert on expected behavior
kubectl logs -n falco daemonset/falco --since=2m | \
  grep -v "INFO\|DEBUG" | tail -20
 
## 4. Check the service is still responding during chaos
for i in $(seq 1 10); do
  STATUS=$(curl -s -o /dev/null -w "%{http_code}" \
    http://payment-service.payment-service.svc.cluster.local:8080/health)
  echo "$(date +%H:%M:%S) Health check: $STATUS"
  sleep 5
done
 
## Collect the final chaos result
kubectl get chaosresult payment-pod-kill-pod-delete \
  -n payment-service -o yaml | grep -A 10 "status:"
 
echo "✅ Security chaos experiment complete"

PYTHON

## chaos/verify-security-controls.py
## After chaos experiments, verify that security controls held
 
import subprocess
import json
import sys
 
def run_kubectl(args):
    """Run a kubectl command and return stdout."""
    result = subprocess.run(
        ["kubectl"] + args.split(),
        capture_output=True, text=True
    )
    return result.stdout, result.returncode
 
 
def check_no_exposed_secrets():
    """Verify no secrets are in pod environment variables."""
    stdout, _ = run_kubectl(
        "get pods -n payment-service -l app=payment-service -o name"
    )
    pods = [p.strip() for p in stdout.strip().split("\n") if p]
 
    for pod in pods:
        pod_name = pod.replace("pod/", "")
        ## Check environment variables for suspicious patterns
        stdout, _ = run_kubectl(
            f"exec -n payment-service {pod_name} -- env"
        )
 
        dangerous_vars = [
            line for line in stdout.split("\n")
            if any(kw in line.lower() for kw in
                   ["password", "secret", "api_key", "token", "credential"])
        ]
 
        if dangerous_vars:
            print(f"❌ FAIL: Pod {pod_name} has sensitive env vars: {dangerous_vars}")
            return False
 
    print("✅ PASS: No secrets in pod environment variables")
    return True
 
 
def check_vault_leases_active():
    """Verify Vault has active leases for the payment service."""
    stdout, _ = run_kubectl(
        "exec -n vault vault-0 -- vault list sys/leases/lookup/database/creds/payment-service"
    )
 
    if "No value found" in stdout or not stdout.strip():
        print("⚠️  WARNING: No active Vault leases found for payment-service")
        return False
 
    lease_count = len([l for l in stdout.strip().split("\n") if l and l != "Keys"])
    print(f"✅ PASS: {lease_count} active Vault lease(s) for payment-service")
    return True
 
 
def check_mtls_active():
    """Verify mTLS is still enforced after chaos."""
    stdout, _ = run_kubectl(
        "get peerauthentication -n payment-service -o jsonpath={.items[0].spec.mtls.mode}"
    )
 
    if stdout.strip() == "STRICT":
        print("✅ PASS: mTLS STRICT mode active on payment-service namespace")
        return True
    else:
        print(f"❌ FAIL: mTLS mode is '{stdout.strip()}' (expected STRICT)")
        return False
 
 
def check_falco_no_critical_alerts():
    """Check Falco for any CRITICAL alerts during the chaos window."""
    stdout, _ = run_kubectl("logs -n falco daemonset/falco --since=5m")
 
    critical_alerts = [
        line for line in stdout.split("\n")
        if "Critical" in line and "Shell Spawned" not in line
        ## Exclude expected test alerts
    ]
 
    if critical_alerts:
        print(f"⚠️  WARNING: Unexpected Falco CRITICAL alerts during chaos:")
        for alert in critical_alerts[:5]:
            print(f"  {alert}")
    else:
        print("✅ PASS: No unexpected Falco CRITICAL alerts during chaos")
 
    return len(critical_alerts) == 0
 
 
if __name__ == "__main__":
    print("=== Security Control Verification After Chaos ===")
    print()
 
    results = [
        check_no_exposed_secrets(),
        check_vault_leases_active(),
        check_mtls_active(),
        check_falco_no_critical_alerts(),
    ]
 
    print()
    if all(results):
        print("✅ ALL CHECKS PASSED — Security controls held under chaos")
        sys.exit(0)
    else:
        failed = sum(1 for r in results if not r)
        print(f"❌ {failed} CHECK(S) FAILED — Review findings above")
        sys.exit(1)

Bash

## Run the verification script after each chaos experiment
python3 chaos/verify-security-controls.py

Component 5 — Unified Compliance Dashboard

One view of your entire security posture

You now have outputs from six different security tools: Gitleaks, Semgrep, Trivy, tfsec, Falco, kube-bench, and the Vault audit log. Each tool produces findings in a different format. A security engineer investigating a potential incident has to check six different places.

The unified dashboard collects findings from all tools, normalizes them to a common schema, and presents them in a single web interface. It also tracks compliance status over time — showing whether your security posture is improving or degrading.

PYTHON

## dashboard/collector.py
## Collects security findings from all tools and normalizes them
 
import json
import subprocess
import os
import requests
from datetime import datetime
from typing import List, Dict
 
 
def normalize_finding(
    source: str,
    severity: str,
    title: str,
    description: str,
    location: str = "",
    cve_id: str = "",
) -> Dict:
    """Normalize a finding from any tool into a common schema."""
    return {
        "id": f"{source}-{hash(title + location) % 100000}",
        "source": source,
        "severity": severity.upper(),
        ## Map tool-specific severities to our standard levels
        "normalized_severity": normalize_severity(source, severity),
        "title": title,
        "description": description,
        "location": location,
        "cve_id": cve_id,
        "timestamp": datetime.utcnow().isoformat() + "Z",
        "status": "open",
    }
 
 
def normalize_severity(source: str, severity: str) -> str:
    """Map tool-specific severity terms to CRITICAL/HIGH/MEDIUM/LOW."""
    severity_map = {
        "CRITICAL": "CRITICAL",
        "HIGH": "HIGH",
        "ERROR": "HIGH",
        "MEDIUM": "MEDIUM",
        "WARNING": "MEDIUM",
        "WARN": "MEDIUM",
        "LOW": "LOW",
        "INFO": "LOW",
        "INFORMATIONAL": "LOW",
    }
    return severity_map.get(severity.upper(), "MEDIUM")
 
 
def collect_trivy_findings(results_file: str) -> List[Dict]:
    """Parse Trivy SARIF output and extract findings."""
    if not os.path.exists(results_file):
        return []
 
    findings = []
    with open(results_file) as f:
        sarif = json.load(f)
 
    for run in sarif.get("runs", []):
        for result in run.get("results", []):
            rule_id = result.get("ruleId", "unknown")
            message = result.get("message", {}).get("text", "")
            severity = result.get("level", "warning")
 
            ## Extract location from the SARIF result
            location = ""
            locations = result.get("locations", [])
            if locations:
                uri = locations[0].get("physicalLocation", {}).get(
                    "artifactLocation", {}
                ).get("uri", "")
                line = locations[0].get("physicalLocation", {}).get(
                    "region", {}
                ).get("startLine", "")
                location = f"{uri}:{line}" if line else uri
 
            findings.append(normalize_finding(
                source="trivy",
                severity=severity,
                title=f"CVE: {rule_id}",
                description=message,
                location=location,
                cve_id=rule_id if rule_id.startswith("CVE-") else "",
            ))
 
    return findings
 
 
def collect_semgrep_findings(results_file: str) -> List[Dict]:
    """Parse Semgrep JSON output and extract findings."""
    if not os.path.exists(results_file):
        return []
 
    findings = []
    with open(results_file) as f:
        semgrep_output = json.load(f)
 
    for result in semgrep_output.get("results", []):
        findings.append(normalize_finding(
            source="semgrep",
            severity=result.get("extra", {}).get("severity", "WARNING"),
            title=result.get("check_id", "unknown-rule"),
            description=result.get("extra", {}).get("message", ""),
            location=f"{result.get('path', '')}:{result.get('start', {}).get('line', '')}",
        ))
 
    return findings
 
 
def collect_tfsec_findings(results_file: str) -> List[Dict]:
    """Parse tfsec JSON output and extract findings."""
    if not os.path.exists(results_file):
        return []
 
    findings = []
    with open(results_file) as f:
        tfsec_output = json.load(f)
 
    for result in tfsec_output.get("results", []):
        findings.append(normalize_finding(
            source="tfsec",
            severity=result.get("severity", "MEDIUM"),
            title=result.get("rule_id", "unknown"),
            description=result.get("description", ""),
            location=f"{result.get('location', {}).get('filename', '')}:"
                     f"{result.get('location', {}).get('start_line', '')}",
        ))
 
    return findings
 
 
def collect_falco_findings() -> List[Dict]:
    """Read recent Falco alerts from pod logs."""
    findings = []
 
    result = subprocess.run(
        ["kubectl", "logs", "-n", "falco", "daemonset/falco", "--since=24h"],
        capture_output=True, text=True,
    )
 
    for line in result.stdout.split("\n"):
        if not line.strip():
            continue
 
        ## Falco log format: TIMESTAMP SEVERITY MESSAGE (key=value ...)
        if "Critical" in line or "Warning" in line or "Error" in line:
            parts = line.split(" ", 2)
            if len(parts) >= 3:
                severity = "HIGH" if "Critical" in parts[1] else "MEDIUM"
                findings.append(normalize_finding(
                    source="falco",
                    severity=severity,
                    title="Runtime security event",
                    description=parts[2] if len(parts) > 2 else line,
                    location="runtime",
                ))
 
    return findings
 
 
def collect_all_findings() -> List[Dict]:
    """Aggregate findings from all security tools."""
    all_findings = []
    all_findings.extend(collect_trivy_findings("trivy-image-results.sarif"))
    all_findings.extend(collect_trivy_findings("trivy-sca-results.sarif"))
    all_findings.extend(collect_semgrep_findings("semgrep-results.json"))
    all_findings.extend(collect_tfsec_findings("tfsec-results.json"))
    all_findings.extend(collect_falco_findings())
    return all_findings
 
 
def calculate_risk_score(findings: List[Dict]) -> Dict:
    """Calculate an overall risk score from all findings."""
    weights = {"CRITICAL": 10, "HIGH": 5, "MEDIUM": 2, "LOW": 1}
    severity_counts = {"CRITICAL": 0, "HIGH": 0, "MEDIUM": 0, "LOW": 0}
 
    total_weight = 0
    for finding in findings:
        sev = finding.get("normalized_severity", "LOW")
        severity_counts[sev] = severity_counts.get(sev, 0) + 1
        total_weight += weights.get(sev, 1)
 
    ## Risk score: 0 (perfect) to 100 (critical)
    ## Penalise CRITICAL findings heavily
    raw_score = min(100, total_weight * 2)
    risk_score = 100 - raw_score
 
    grade = "A" if risk_score >= 90 else \
            "B" if risk_score >= 75 else \
            "C" if risk_score >= 60 else \
            "D" if risk_score >= 40 else "F"
 
    return {
        "score": max(0, risk_score),
        "grade": grade,
        "severity_counts": severity_counts,
        "total_findings": len(findings),
    }

PYTHON

## dashboard/app.py
## Flask web application for the compliance dashboard
 
from flask import Flask, render_template_string, jsonify
from dashboard.collector import collect_all_findings, calculate_risk_score
 
app = Flask(__name__)
 
DASHBOARD_HTML = """
<!DOCTYPE html>
<html>
<head>
    <title>DevSecOps Compliance Dashboard — Razorpay</title>
    <style>
        body { font-family: monospace; background: #0d1117; color: #c9d1d9; margin: 2rem; }
        h1 { color: #58a6ff; border-bottom: 1px solid #30363d; padding-bottom: 1rem; }
        .score-card { background: #161b22; border: 1px solid #30363d; padding: 1.5rem;
                      border-radius: 6px; display: inline-block; margin: 1rem 1rem 1rem 0; }
        .grade-A { color: #3fb950; } .grade-B { color: #56d364; }
        .grade-C { color: #e3b341; } .grade-D { color: #f85149; } .grade-F { color: #f85149; }
        .critical { color: #f85149; } .high { color: #e3b341; }
        .medium { color: #d29922; } .low { color: #8b949e; }
        table { border-collapse: collapse; width: 100%; margin-top: 1rem; }
        th, td { padding: 0.5rem 1rem; text-align: left;
                 border-bottom: 1px solid #30363d; }
        th { background: #161b22; color: #58a6ff; }
        .source-badge { background: #21262d; padding: 2px 8px;
                        border-radius: 12px; font-size: 0.85em; }
    </style>
</head>
<body>
    <h1>🔐 DevSecOps Compliance Dashboard</h1>
 
    <div class="score-card">
        <div style="font-size: 3rem; font-weight: bold;"
             class="grade-{{ risk.grade }}">{{ risk.grade }}</div>
        <div>Security Score: {{ risk.score }}/100</div>
        <div>Total Findings: {{ risk.total_findings }}</div>
    </div>
 
    <div class="score-card">
        <div><span class="critical">● CRITICAL: {{ risk.severity_counts.CRITICAL }}</span></div>
        <div><span class="high">● HIGH: {{ risk.severity_counts.HIGH }}</span></div>
        <div><span class="medium">● MEDIUM: {{ risk.severity_counts.MEDIUM }}</span></div>
        <div><span class="low">● LOW: {{ risk.severity_counts.LOW }}</span></div>
    </div>
 
    <h2>All Findings</h2>
    <table>
        <tr>
            <th>Severity</th><th>Source</th><th>Title</th>
            <th>Location</th><th>Timestamp</th>
        </tr>
        {% for finding in findings|sort(attribute='normalized_severity') %}
        <tr>
            <td class="{{ finding.normalized_severity|lower }}">
                {{ finding.normalized_severity }}
            </td>
            <td><span class="source-badge">{{ finding.source }}</span></td>
            <td>{{ finding.title }}</td>
            <td style="font-size: 0.85em;">{{ finding.location }}</td>
            <td style="font-size: 0.85em;">{{ finding.timestamp[:19] }}</td>
        </tr>
        {% endfor %}
    </table>
</body>
</html>
"""
 
@app.route("/")
def dashboard():
    findings = collect_all_findings()
    risk = calculate_risk_score(findings)
    return render_template_string(DASHBOARD_HTML, findings=findings, risk=risk)
 
@app.route("/api/findings")
def api_findings():
    findings = collect_all_findings()
    risk = calculate_risk_score(findings)
    return jsonify({"findings": findings, "risk_summary": risk})
 
if __name__ == "__main__":
    app.run(host="0.0.0.0", port=5000, debug=False)

Bash

## Run the dashboard
pip3 install flask --break-system-packages
python3 dashboard/app.py &
 
## Access at http://localhost:5000
curl http://localhost:5000/api/findings | python3 -m json.tool | head -40

Production Checklist

Bash

## ─── IaC Scanning ────────────────────────────────────────────
tfsec modules/ --minimum-severity HIGH --exit-code 1 --quiet
echo "✅ No HIGH/CRITICAL IaC misconfigurations"
 
checkov -d . --framework terraform --compact --quiet
echo "✅ Checkov IaC scan passed"
 
## ─── Vault ───────────────────────────────────────────────────
## Verify Vault is unsealed and accessible
kubectl exec -n vault vault-0 -- vault status | grep "Sealed"
## Expected: Sealed false
 
## Verify dynamic credentials are being issued
kubectl exec -n vault vault-0 -- \
  vault read database/creds/payment-service
## Expected: username and password fields with values
 
## Verify no static secrets in Kubernetes Secrets
kubectl get secrets -n payment-service \
  -o jsonpath='{.items[*].metadata.name}'
## Should show NO database password secrets
 
## ─── Istio mTLS ───────────────────────────────────────────────
istioctl x describe pod \
  $(kubectl get pods -n payment-service -l app=payment-service -o name | head -1) \
  -n payment-service | grep -i "mtls\|strict"
## Expected: mTLS is STRICT
 
kubectl get peerauthentication -n payment-service \
  -o jsonpath='{.items[0].spec.mtls.mode}'
## Expected: STRICT
 
## ─── Chaos Engineering ───────────────────────────────────────
python3 chaos/verify-security-controls.py
## Expected: ALL CHECKS PASSED
 
## ─── Dashboard ───────────────────────────────────────────────
curl -s http://localhost:5000/api/findings | \
  python3 -c "
import json, sys
data = json.load(sys.stdin)
risk = data['risk_summary']
print(f'Security Grade: {risk[\"grade\"]} ({risk[\"score\"]}/100)')
print(f'CRITICAL findings: {risk[\"severity_counts\"][\"CRITICAL\"]}')
assert risk['severity_counts']['CRITICAL'] == 0, 'CRITICAL findings must be zero'
print('✅ Dashboard reporting zero CRITICAL findings')
"
 
echo ""
echo "✅ All Platform Security components verified"

Common Production Mistakes

Running tfsec and Checkov only on the main branch after merge. IaC misconfigurations caught after merge to main are misconfigurations that have already been reviewed and approved by your team. Run IaC scanning on every pull request as a blocking check, not just on the main branch pipeline. The earlier a misconfiguration is caught, the less expensive it is to fix — catching it before review means no developer has read and approved insecure infrastructure.

Using Vault in dev mode for anything beyond local development. Dev mode stores everything in memory — a pod restart loses all secrets, all leases, all configuration. Vault for production requires a persistent storage backend (integrated storage or Consul), an HA deployment with at least 3 nodes, a proper unseal strategy (cloud KMS auto-unseal), and a documented recovery procedure. Never let a dev mode Vault deployment persist beyond the day you set it up.

Not configuring Vault lease renewal. When Vault issues a dynamic database credential with a 1-hour TTL, the application must renew the lease before it expires or the credential becomes invalid and the application cannot connect to the database. The Vault Agent sidecar handles renewal automatically. If you are using the SDK directly, implement the lease renewal loop explicitly — a missed renewal causes a silent database connection failure at an unpredictable time.

Enabling Istio mTLS in STRICT mode on namespaces that still have non-mesh pods. If any pod in the namespace does not have an Envoy sidecar (because it was created before istio-injection=enabled was set), all traffic to that pod is rejected. Roll out mTLS in PERMISSIVE mode first, verify all pods have sidecars with kubectl get pods -n NAMESPACE -o jsonpath='{.items[*].spec.containers[*].name}', then switch to STRICT. Use istioctl analyze to catch configuration issues before they cause outages.

Running chaos experiments without a defined hypothesis. Chaos engineering without a hypothesis is just random breakage. Before running any experiment, write down: "I believe that when X happens, Y security control will hold and Z will be the observable evidence." Without a hypothesis, you cannot know whether the experiment revealed a gap or just confirmed existing behavior. The experiment from this capstone should have a hypothesis like: "When 50% of payment-service pods are killed, Vault will reinject fresh credentials to replacement pods within 90 seconds, and no credentials will appear in pod environment variables."

Treating the compliance dashboard as a compliance checkbox. A dashboard that shows "Grade A" is only meaningful if the underlying data is fresh and complete. Schedule a nightly CI job that re-collects all findings, updates the dashboard data, and alerts on any degradation from the previous day's score. A static dashboard that shows last week's results while this week's deployment introduced three critical CVEs is worse than no dashboard — it provides false assurance.

Quick Reference

Component	Tool	What It Protects	Key Config File
IaC Scanning	tfsec + Checkov	Infrastructure misconfig	.tfsec.toml
Secrets	HashiCorp Vault	Dynamic database credentials	vault policy
Service Mesh	Istio	mTLS between services	PeerAuthentication
Chaos Testing	LitmusChaos	Validates controls under failure	ChaosEngine YAML
Dashboard	Flask + collectors	Unified security posture view	dashboard/app.py

Command	What It Does
`tfsec modules/ --minimum-severity HIGH`	Scan Terraform for HIGH+ misconfigs
`vault read database/creds/payment-service`	Generate a dynamic DB credential
`vault lease revoke -prefix database/creds/payment-service`	Revoke all active leases
`istioctl x describe pod POD_NAME -n NAMESPACE`	Check mTLS status of a pod
`kubectl apply -f chaos/pod-kill-experiment.yaml`	Run pod kill chaos experiment
`python3 chaos/verify-security-controls.py`	Verify controls after chaos
`curl http://localhost:5000/api/findings`	Get all findings as JSON

Videos & Guides

No external walkthroughs, video tutorials, or reference files attached to this blueprint yet.