DevSecOps project 3 — The Platform Security Challenge | Engineering Modules | DevOps Network

Engineering ModuleADVANCED

DevSecOps project 3 — The Platform Security Challenge

The mega-capstone. Wire Terraform IaC scanning, secrets management with Vault, zero-trust mTLS, security chaos engineering, and a unified compliance dashboard into one production security platform.

~150 Minutes

10 Topics

Hands-on Scenarios

What You'll Learn

The Question That Exposes Gaps

A CISO at a fintech startup walked into the engineering team's standup and asked one question: "If I randomly terminate 20% of our containers right now, will any secrets get exposed?" Nobody could answer. They had secrets in Kubernetes Secrets (base64-encoded, not encrypted). They had TLS at the ingress but plaintext traffic between internal services. They had security tools installed but no centralized view of their security posture. They had infrastructure as code but no scanning to ensure it stayed secure. Each of those gaps is a separate problem. This capstone treats them as one problem: building a cohesive security platform where every piece reinforces the others. You are building five components that integrate into a unified platform: ┌────────────────────────────────────────────────────────────┐ │ Component 1: Terraform Security (tfsec + Checkov) │ │ Scan IaC before provisioning — catch misconfigs at source │ ├────────────────────────────────────────────────────────────┤ │ Component 2: HashiCorp Vault │ │ Dynamic secrets — no static credentials anywhere │ ├────────────────────────────────────────────────────────────┤ │ Component 3: Istio mTLS + Zero-Trust Networking │ │ Every service-to-service call encrypted and authenticated │ ├────────────────────────────────────────────────────────────┤ │ Component 4: Security Chaos Engineering (Chaos Monkey) │ │ Deliberately break things to test security controls │ ├────────────────────────────────────────────────────────────┤ │ Component 5: Unified Compliance Dashboard │ │ Single pane of glass across all security tools │ └────────────────────────────────────────────────────────────┘

Component 1 — Infrastructure as Code Security

### Why IaC misconfigurations are expensive to fix later Infrastructure as Code is both a security strength and a security risk. The strength: every infrastructure change is in version control, reviewable, auditable. The risk: one misconfigured Terraform module can provision hundreds of insecure resources across all environments simultaneously. An S3 bucket with `acl = "public-read"` in a Terraform module gets deployed to dev, staging, and production in the same `terraform apply`. If the misconfiguration is not caught in the CI pipeline, it sits there until someone notices a billing anomaly or — worse — a breach. **tfsec** and **Checkov** scan Terraform code before it is applied. They catch misconfigurations that are expensive to fix after provisioning: S3 buckets without server-side encryption, security groups open to 0.0.0.0/0, EKS clusters without private API endpoints, RDS instances without deletion protection. ### Writing the infrastructure code ```bash ## Set up the IaC project structure mkdir terraform-infrastructure && cd terraform-infrastructure mkdir -p modules/eks modules/rds modules/vpc modules/s3 environments/production touch main.tf variables.tf outputs.tf touch modules/eks/main.tf modules/rds/main.tf echo "✅ IaC project structure created" ``` ```hcl ## modules/eks/main.tf ## EKS cluster configuration ## This file intentionally contains misconfigurations that the scanner catches terraform { required_providers { aws = { source = "hashicorp/aws" version = "~> 5.0" } } } variable "cluster_name" { type = string description = "Name of the EKS cluster" } variable "cluster_version" { type = string description = "Kubernetes version for the cluster" default = "1.29" } ## MISCONFIGURATION 1: Public API endpoint enabled ## tfsec will flag: aws-eks-no-public-cluster-access ## The Kubernetes API server should not be publicly accessible resource "aws_eks_cluster" "main" { name = var.cluster_name version = var.cluster_version role_arn = aws_iam_role.eks_cluster.arn vpc_config { subnet_ids = var.subnet_ids ## ❌ Public endpoint should be false in production endpoint_public_access = true endpoint_private_access = true ## ❌ API access is open to all IPs — should be restricted to VPN CIDR public_access_cidrs = ["0.0.0.0/0"] } ## MISCONFIGURATION 2: Secrets not encrypted with KMS ## tfsec will flag: aws-eks-enable-control-plane-logging ## encryption_config block is missing entirely } ## MISCONFIGURATION 3: CloudWatch logging not enabled ## tfsec will flag: aws-eks-enable-control-plane-logging resource "aws_eks_cluster" "logging_disabled" { name = "${var.cluster_name}-bad" enabled_cluster_log_types = [] ## ❌ should include ["api", "audit", "authenticator"] } ``` ```hcl ## modules/rds/main.tf ## RDS database configuration with deliberate misconfigurations variable "db_identifier" { type = string } ## MISCONFIGURATION 4: RDS instance publicly accessible ## tfsec will flag: aws-rds-no-public-db-access resource "aws_db_instance" "payment_db" { identifier = var.db_identifier engine = "postgres" engine_version = "15.4" ## ❌ Production databases should never be publicly accessible publicly_accessible = true ## ❌ Deletion protection prevents accidental deletion in production deletion_protection = false ## ❌ Automated backups provide point-in-time recovery backup_retention_period = 0 storage_encrypted = true ## This one is correct ✅ } ``` ### Running the IaC security scans ```bash ## ── tfsec: deep Terraform security analysis ────────────────── tfsec modules/ --format=json --out=tfsec-results.json ## View high-severity findings only tfsec modules/ --minimum-severity HIGH ## Expected output: ## ## Result #1 HIGH ## ────────────────────────────────────────────────────────────── ## ID aws-eks-no-public-cluster-access ## Impact EKS Kubernetes API server accessible to any IP ## Resolution Restrict public access CIDRs to specific ranges ## More Info https://aquasecurity.github.io/tfsec/... ## ## modules/eks/main.tf:22-34 ## 22 resource "aws_eks_cluster" "main" { ## ... ## 30 public_access_cidrs = ["0.0.0.0/0"] ## ## Passed: 3 Failed: 4 Ignored: 0 ``` ```bash ## ── Checkov: multi-framework IaC scanning ───────────────────── ## Checkov checks Terraform, Kubernetes YAML, Dockerfile, CloudFormation checkov -d . \ --framework terraform \ --output json \ --output-file-path checkov-results.json \ --soft-fail ## run without blocking — generate report first ## Count failures by severity python3 -c " import json with open('checkov-results.json') as f: results = json.load(f) checks = results.get('results', {}) failed = checks.get('failed_checks', []) passed = checks.get('passed_checks', []) severity_count = {} for check in failed: sev = check.get('severity', 'UNKNOWN') severity_count[sev] = severity_count.get(sev, 0) + 1 print(f'Total passed: {len(passed)}') print(f'Total failed: {len(failed)}') print() print('Failed by severity:') for sev, count in sorted(severity_count.items()): print(f' {sev}: {count}') " ``` ### Fixing the IaC misconfigurations ```hcl ## modules/eks/main.tf — SECURE VERSION resource "aws_eks_cluster" "main" { name = var.cluster_name version = var.cluster_version role_arn = aws_iam_role.eks_cluster.arn ## ✅ Enable all audit logging enabled_cluster_log_types = ["api", "audit", "authenticator", "controllerManager", "scheduler"] vpc_config { subnet_ids = var.subnet_ids ## ✅ Private only — API server not exposed to internet endpoint_public_access = false endpoint_private_access = true } ## ✅ Encrypt Kubernetes secrets in etcd with KMS encryption_config { provider { key_arn = aws_kms_key.eks_secrets.arn } resources = ["secrets"] } } ## ✅ Dedicated KMS key for EKS secret encryption resource "aws_kms_key" "eks_secrets" { description = "KMS key for EKS secrets encryption" deletion_window_in_days = 7 enable_key_rotation = true ## rotate the key annually tags = { Name = "${var.cluster_name}-eks-secrets" Environment = "production" ManagedBy = "terraform" } } ``` ```hcl ## modules/rds/main.tf — SECURE VERSION resource "aws_db_instance" "payment_db" { identifier = var.db_identifier engine = "postgres" engine_version = "15.4" ## ✅ Never publicly accessible publicly_accessible = false ## ✅ Prevent accidental deletion — disable only for deliberate teardown deletion_protection = true ## ✅ 7 days of automated backups for point-in-time recovery backup_retention_period = 7 ## ✅ Encrypt data at rest storage_encrypted = true kms_key_id = aws_kms_key.rds.arn ## ✅ Enable enhanced monitoring and Performance Insights monitoring_interval = 60 performance_insights_enabled = true } ``` ```bash ## Add tfsec to the CI/CD pipeline from Capstone 1 ## Add this as the first job in secure-pipeline.yml ## (after secrets-detection, before sast-scan) cat >> .github/workflows/secure-pipeline.yml << 'EOF' ## ── IaC Security Scan (add after secrets-detection) ────────── iac-scan: name: "Gate 0.5 — IaC Security Scan (tfsec + Checkov)" runs-on: ubuntu-latest needs: secrets-detection steps: - name: Checkout code uses: actions/checkout@v4 - name: Run tfsec uses: aquasecurity/tfsec-action@v1.0.3 with: working_directory: terraform-infrastructure/ severity: HIGH soft_fail: false - name: Run Checkov uses: bridgecrewio/checkov-action@master with: directory: terraform-infrastructure/ framework: terraform soft_fail: false output_format: sarif output_file_path: checkov-results.sarif EOF echo "✅ IaC scanning added to pipeline" ```

Component 2 — HashiCorp Vault for Dynamic Secrets

### Why static secrets are the root cause of most breaches Every rotation schedule for static secrets has the same failure mode: the rotation does not happen on schedule because it breaks something, and "temporary" static secrets end up permanent for months or years. **HashiCorp Vault** eliminates static secrets by making credentials dynamic. Instead of your application knowing the database password, Vault generates a unique database credential for each application instance, valid for 1 hour. When the lease expires, Vault automatically revokes it. A leaked credential is useless within an hour. ```bash ## Install Vault in development mode for this capstone ## Production Vault uses HA configuration with cloud storage helm repo add hashicorp https://helm.releases.hashicorp.com ## Install Vault in dev mode (single node, in-memory storage) ## Replace with production HA config for real deployments helm install vault hashicorp/vault \ --namespace vault \ --create-namespace \ --set "server.dev.enabled=true" \ --set "injector.enabled=true" ## Wait for Vault to be ready kubectl rollout status deployment/vault -n vault --timeout=120s ## Port-forward for initial configuration kubectl port-forward svc/vault 8200:8200 -n vault & export VAULT_ADDR="http://localhost:8200" export VAULT_TOKEN="root" ## dev mode uses "root" as the token echo "✅ Vault installed" ``` ### Configuring Vault for the payment service ```bash ## Enable the PostgreSQL database secrets engine vault secrets enable database ## Configure the PostgreSQL connection ## Vault connects to the database with admin credentials ## and creates/revokes credentials on behalf of applications vault write database/config/payment-db \ plugin_name=postgresql-database-plugin \ allowed_roles="payment-service" \ connection_url="postgresql://{{username}}:{{password}}@postgres.database.svc:5432/payments" \ username="vault_admin" \ password="$(kubectl get secret postgres-admin -n database -o jsonpath='{.data.password}' | base64 -d)" ## Create a role that defines what credentials Vault generates vault write database/roles/payment-service \ db_name=payment-db \ creation_statements=" CREATE ROLE \"{{name}}\" WITH LOGIN PASSWORD '{{password}}' VALID UNTIL '{{expiration}}'; GRANT SELECT, INSERT, UPDATE ON payments TO \"{{name}}\"; GRANT SELECT ON merchants TO \"{{name}}\"; " \ revocation_statements=" REVOKE ALL ON ALL TABLES IN SCHEMA public FROM \"{{name}}\"; DROP ROLE IF EXISTS \"{{name}}\"; " \ ## Credentials valid for 1 hour — Vault renews automatically until max_ttl default_ttl="1h" \ max_ttl="24h" ## Test: generate a credential manually vault read database/creds/payment-service ## Output: ## Key Value ## --- ----- ## lease_id database/creds/payment-service/KX... ## lease_duration 1h ## username v-payment-KoiE8L8e ## password A1B-random-generated-credential echo "✅ Database secrets engine configured" ``` ### Kubernetes auth and the Vault Agent Injector **Vault Agent Injector** uses a Kubernetes mutating webhook to inject a sidecar container into pods. The sidecar authenticates to Vault using the pod's ServiceAccount token and writes secrets to a shared volume. The application reads secrets from files — no code changes needed. ```bash ## Enable Kubernetes authentication in Vault vault auth enable kubernetes ## Configure Kubernetes auth with the cluster's details vault write auth/kubernetes/config \ kubernetes_host="https://$(kubectl get svc kubernetes -o jsonpath='{.spec.clusterIP}'):443" \ kubernetes_ca_cert=@<(kubectl get secret \ $(kubectl get sa vault -n vault -o jsonpath='{.secrets[0].name}') \ -n vault -o jsonpath='{.data.ca\.crt}' | base64 -d) \ token_reviewer_jwt="$(kubectl create token vault -n vault)" ## Create a Vault policy for the payment service ## This policy says: the payment service can read its own database credentials vault policy write payment-service - << 'EOF' path "database/creds/payment-service" { capabilities = ["read"] } path "kv/data/payment-service/*" { capabilities = ["read"] } EOF ## Bind the Kubernetes ServiceAccount to the Vault policy vault write auth/kubernetes/role/payment-service \ bound_service_account_names=payment-service \ bound_service_account_namespaces=payment-service \ policies=payment-service \ ttl=1h echo "✅ Vault Kubernetes auth configured" ``` ```yaml ## deployments/payment-service-with-vault.yaml ## The Vault Agent Injector annotations tell the webhook to inject ## a sidecar that fetches credentials from Vault apiVersion: apps/v1 kind: Deployment metadata: name: payment-service namespace: payment-service spec: replicas: 2 selector: matchLabels: app: payment-service template: metadata: labels: app: payment-service annotations: ## These annotations trigger the Vault Agent Injector webhook vault.hashicorp.com/agent-inject: "true" vault.hashicorp.com/role: "payment-service" ## Inject the database credentials as a file at /vault/secrets/db-creds vault.hashicorp.com/agent-inject-secret-db-creds: "database/creds/payment-service" ## Template the credentials into a format the app can source vault.hashicorp.com/agent-inject-template-db-creds: | {{- with secret "database/creds/payment-service" -}} export DB_USERNAME="{{ .Data.username }}" export DB_PASSWORD="{{ .Data.password }}" {{- end }} spec: serviceAccountName: payment-service containers: - name: payment-service image: ghcr.io/razorpay/payment-service:abc1234 command: ["/bin/sh", "-c"] args: ## Source the Vault-injected credentials before starting the app - | source /vault/secrets/db-creds python src/app.py env: - name: DB_HOST value: "postgres.database.svc.cluster.local" - name: DB_PORT value: "5432" ## DB_USERNAME and DB_PASSWORD come from Vault via /vault/secrets/db-creds securityContext: allowPrivilegeEscalation: false readOnlyRootFilesystem: true runAsNonRoot: true runAsUser: 10001 capabilities: drop: ["ALL"] resources: requests: cpu: "100m" memory: "128Mi" limits: cpu: "500m" memory: "256Mi" volumeMounts: - name: tmp-dir mountPath: /tmp volumes: - name: tmp-dir emptyDir: {} ``` ```bash ## Verify Vault injection is working kubectl apply -f deployments/payment-service-with-vault.yaml kubectl rollout status deployment/payment-service -n payment-service ## Check that credentials were injected kubectl exec -n payment-service \ $(kubectl get pods -n payment-service -l app=payment-service -o name | head -1) \ -c payment-service \ -- cat /vault/secrets/db-creds ## Expected output: ## export DB_USERNAME="v-payment-KoiE8L8e" ## export DB_PASSWORD="A1B-xYz-random-2024" echo "✅ Vault dynamic secrets working" ```

Component 3 — Istio mTLS and Zero-Trust Networking

### The problem with trusting internal traffic Zero-trust security means "never trust, always verify" — including traffic from inside your own network. In a traditional Kubernetes setup without service mesh, if pod A can reach pod B on port 8080, there is no authentication happening at the transport layer. Pod B trusts the traffic because it comes from inside the cluster. **mTLS (mutual TLS)** fixes this: both sides of every connection present certificates and verify each other's identity. Pod A proves it is allowed to talk to pod B. Pod B proves it is the real payment service, not an impersonating pod. **Istio** is a service mesh that implements mTLS transparently using sidecar proxies (Envoy). Applications do not change. The Envoy sidecar handles all certificate management, rotation, and verification automatically. ```bash ## Install Istio curl -L https://istio.io/downloadIstio | \ ISTIO_VERSION=1.20.1 TARGET_ARCH=x86_64 sh - export PATH="$PWD/istio-1.20.1/bin:$PATH" ## Install Istio in the cluster with default profile ## Default = pilot (control plane) + ingress gateway istioctl install --set profile=default -y ## Verify Istio is running kubectl get pods -n istio-system ## Expected: istiod and istio-ingressgateway in Running state echo "✅ Istio installed" ``` ```bash ## Enable Istio sidecar injection for the payment-service namespace ## This label tells Istio to automatically inject Envoy sidecar into all new pods kubectl label namespace payment-service istio-injection=enabled ## Restart pods to get sidecars injected kubectl rollout restart deployment/payment-service -n payment-service ## Verify sidecar was injected — each pod should have 2 containers now kubectl get pods -n payment-service ## Expected: READY 2/2 (app container + envoy sidecar) ``` ```yaml ## istio/peer-authentication.yaml ## PeerAuthentication enforces mTLS for all traffic within the namespace ## STRICT mode means plaintext connections are rejected — mTLS only apiVersion: security.istio.io/v1beta1 kind: PeerAuthentication metadata: name: default namespace: payment-service spec: ## mtls.mode: STRICT = reject all non-mTLS connections ## This means even other pods without Istio sidecars cannot connect mtls: mode: STRICT ``` ```yaml ## istio/authorization-policy.yaml ## AuthorizationPolicy controls which services can communicate ## This is layer 7 (HTTP) access control on top of mTLS apiVersion: security.istio.io/v1beta1 kind: AuthorizationPolicy metadata: name: payment-service-policy namespace: payment-service spec: selector: matchLabels: app: payment-service action: ALLOW rules: ## Only allow traffic from the ingress gateway (external customers) - from: - source: ## The principal is the mTLS identity of the caller ## This format: cluster.local/ns/NAMESPACE/sa/SERVICEACCOUNT principals: - "cluster.local/ns/istio-system/sa/istio-ingressgateway-service-account" to: - operation: methods: ["GET", "POST"] paths: ["/payment/*", "/health"] ``` ```yaml ## istio/deny-all-default.yaml ## Start with deny-all and add explicit allows ## Any traffic not matching an ALLOW policy is denied apiVersion: security.istio.io/v1beta1 kind: AuthorizationPolicy metadata: name: deny-all namespace: payment-service spec: ## No rules = deny all traffic {} ``` ```bash ## Apply all Istio policies kubectl apply -f istio/ ## Test that mTLS is enforced ## A pod without a sidecar should be rejected kubectl run no-sidecar-test \ --image=curlimages/curl:8.4.0 \ --restart=Never \ --namespace=default \ -- curl -s --connect-timeout 3 \ http://payment-service.payment-service.svc.cluster.local:8080/health kubectl logs no-sidecar-test ## Expected: connection refused or RBAC denied (mTLS strict mode rejects plaintext) ## Verify mTLS is active from within the mesh istioctl x describe pod \ $(kubectl get pods -n payment-service -l app=payment-service -o name | head -1) \ -n payment-service | grep -i mtls ## Expected: mTLS is STRICT echo "✅ Istio mTLS zero-trust networking active" ```

Component 4 — Security Chaos Engineering

### Testing your controls by breaking things Security chaos engineering is the practice of deliberately injecting security failures into your systems to verify that controls work and alerts fire. Regular chaos experiments answer the question: "Does our security posture actually hold up under realistic conditions, or do we just think it does?" The approach uses **LitmusChaos** — a Kubernetes-native chaos engineering platform — to inject failures, and then verifies that: * Falco detected the suspicious activity * The application recovered cleanly * No secrets were exposed during the chaos * mTLS prevented unauthorized connections during the disruption ```bash ## Install LitmusChaos kubectl apply -f https://litmuschaos.github.io/litmus/litmus-operator-v3.5.0.yaml ## Wait for LitmusChaos to be ready kubectl rollout status deployment/chaos-operator-ce \ -n litmus --timeout=120s ## Install chaos experiments library kubectl apply -f \ https://hub.litmuschaos.io/api/chaos/3.5.0?file=charts/generic/experiments.yaml \ -n payment-service echo "✅ LitmusChaos installed" ``` ```yaml ## chaos/pod-kill-experiment.yaml ## Chaos Experiment 1: Kill random pods in the payment namespace ## Validates: pod recovery time, no secret exposure during restart, ## Vault reinjects credentials to new pods successfully apiVersion: litmuschaos.io/v1alpha1 kind: ChaosEngine metadata: name: payment-pod-kill namespace: payment-service spec: appinfo: appns: payment-service applabel: "app=payment-service" appkind: deployment chaosServiceAccount: litmus-admin experiments: - name: pod-delete spec: components: env: ## Kill 1 pod every 10 seconds for 60 seconds total - name: TOTAL_CHAOS_DURATION value: "60" - name: CHAOS_INTERVAL value: "10" ## Kill 50% of the matching pods at each interval - name: FORCE value: "false" ``` ```yaml ## chaos/network-loss-experiment.yaml ## Chaos Experiment 2: Introduce 50% packet loss to the payment service ## Validates: service resilience, error rates stay below SLA, ## Falco does not fire false positives on slow connections apiVersion: litmuschaos.io/v1alpha1 kind: ChaosEngine metadata: name: payment-network-loss namespace: payment-service spec: appinfo: appns: payment-service applabel: "app=payment-service" appkind: deployment chaosServiceAccount: litmus-admin experiments: - name: pod-network-loss spec: components: env: - name: TOTAL_CHAOS_DURATION value: "60" ## 50% packet loss — service should still respond (slowly) - name: NETWORK_PACKET_LOSS_PERCENTAGE value: "50" ## Target only egress traffic to the database - name: DESTINATION_IPS value: "$(kubectl get svc postgresql -n database -o jsonpath='{.spec.clusterIP}')" ``` ```bash ## Run the security chaos experiment kubectl apply -f chaos/pod-kill-experiment.yaml ## Monitor the experiment status kubectl get chaosresult payment-pod-kill-pod-delete \ -n payment-service -o jsonpath='{.status.verdict}' ## While chaos runs — verify these in parallel: ## 1. Check Vault is reinjecting credentials to new pods kubectl logs -n vault -l app.kubernetes.io/name=vault-agent-injector \ --since=2m | grep "renewed\|injected" ## 2. Check no secrets are in environment variables (they should be in files) kubectl exec -n payment-service \ $(kubectl get pods -n payment-service -l app=payment-service -o name | head -1) \ -- env | grep -i "password\|secret\|token\|key" ## Expected: NO matches — secrets are in /vault/secrets/, not env vars ## 3. Check Falco did not alert on expected behavior kubectl logs -n falco daemonset/falco --since=2m | \ grep -v "INFO\|DEBUG" | tail -20 ## 4. Check the service is still responding during chaos for i in $(seq 1 10); do STATUS=$(curl -s -o /dev/null -w "%{http_code}" \ http://payment-service.payment-service.svc.cluster.local:8080/health) echo "$(date +%H:%M:%S) Health check: $STATUS" sleep 5 done ## Collect the final chaos result kubectl get chaosresult payment-pod-kill-pod-delete \ -n payment-service -o yaml | grep -A 10 "status:" echo "✅ Security chaos experiment complete" ``` ```python ## chaos/verify-security-controls.py ## After chaos experiments, verify that security controls held import subprocess import json import sys def run_kubectl(args): """Run a kubectl command and return stdout.""" result = subprocess.run( ["kubectl"] + args.split(), capture_output=True, text=True ) return result.stdout, result.returncode def check_no_exposed_secrets(): """Verify no secrets are in pod environment variables.""" stdout, _ = run_kubectl( "get pods -n payment-service -l app=payment-service -o name" ) pods = [p.strip() for p in stdout.strip().split("\n") if p] for pod in pods: pod_name = pod.replace("pod/", "") ## Check environment variables for suspicious patterns stdout, _ = run_kubectl( f"exec -n payment-service {pod_name} -- env" ) dangerous_vars = [ line for line in stdout.split("\n") if any(kw in line.lower() for kw in ["password", "secret", "api_key", "token", "credential"]) ] if dangerous_vars: print(f"❌ FAIL: Pod {pod_name} has sensitive env vars: {dangerous_vars}") return False print("✅ PASS: No secrets in pod environment variables") return True def check_vault_leases_active(): """Verify Vault has active leases for the payment service.""" stdout, _ = run_kubectl( "exec -n vault vault-0 -- vault list sys/leases/lookup/database/creds/payment-service" ) if "No value found" in stdout or not stdout.strip(): print("⚠️ WARNING: No active Vault leases found for payment-service") return False lease_count = len([l for l in stdout.strip().split("\n") if l and l != "Keys"]) print(f"✅ PASS: {lease_count} active Vault lease(s) for payment-service") return True def check_mtls_active(): """Verify mTLS is still enforced after chaos.""" stdout, _ = run_kubectl( "get peerauthentication -n payment-service -o jsonpath={.items[0].spec.mtls.mode}" ) if stdout.strip() == "STRICT": print("✅ PASS: mTLS STRICT mode active on payment-service namespace") return True else: print(f"❌ FAIL: mTLS mode is '{stdout.strip()}' (expected STRICT)") return False def check_falco_no_critical_alerts(): """Check Falco for any CRITICAL alerts during the chaos window.""" stdout, _ = run_kubectl("logs -n falco daemonset/falco --since=5m") critical_alerts = [ line for line in stdout.split("\n") if "Critical" in line and "Shell Spawned" not in line ## Exclude expected test alerts ] if critical_alerts: print(f"⚠️ WARNING: Unexpected Falco CRITICAL alerts during chaos:") for alert in critical_alerts[:5]: print(f" {alert}") else: print("✅ PASS: No unexpected Falco CRITICAL alerts during chaos") return len(critical_alerts) == 0 if __name__ == "__main__": print("=== Security Control Verification After Chaos ===") print() results = [ check_no_exposed_secrets(), check_vault_leases_active(), check_mtls_active(), check_falco_no_critical_alerts(), ] print() if all(results): print("✅ ALL CHECKS PASSED — Security controls held under chaos") sys.exit(0) else: failed = sum(1 for r in results if not r) print(f"❌ {failed} CHECK(S) FAILED — Review findings above") sys.exit(1) ``` ```bash ## Run the verification script after each chaos experiment python3 chaos/verify-security-controls.py ```

DevSecOps project 3 — The Platform Security Challenge

What You'll Learn

The Question That Exposes Gaps

Before You Start

Component 1 — Infrastructure as Code Security

Component 2 — HashiCorp Vault for Dynamic Secrets

Component 3 — Istio mTLS and Zero-Trust Networking

Component 4 — Security Chaos Engineering

Skills You'll Master

Curriculum Index

The Question That Exposes Gaps

Before You Start

Component 1 — Infrastructure as Code Security

Component 2 — HashiCorp Vault for Dynamic Secrets

Component 3 — Istio mTLS and Zero-Trust Networking

Component 4 — Security Chaos Engineering

Component 5 — Unified Compliance Dashboard

Production Checklist

Common Production Mistakes

Quick Reference

Career Impact

Troubleshooting & Lab Guides

How to Troubleshoot a Linux Production Server: A Systematic Approach

Next Modules

Senior DevOps Engineer Interview Prep

S3 to FSx for OpenZFS

CI/CD and Secure Pipeline Design

Related Guides

Terraform State at Scale: Remote Backends, Locking, and Drift in Multi-Team Orgs

Shift-Left Security: Adding SBOM and Supply Chain Scanning to Your CI/CD Pipeline