Your startup just won a contract with a large enterprise. Their procurement team sends a security questionnaire with 180 questions. Your CTO forwards it to you. You have two weeks to answer it, provide evidence, and get an auditor to sign off. You open the questionnaire. Question 14: "Do you enforce multi-factor authentication on all production systems?" You do. But do you have evidence that you did it consistently for the last six months? No. Question 47: "Is all sensitive data encrypted at rest?" Yes — but who configured the S3 bucket encryption? When? Can you show the auditor the policy that enforced it? This is why compliance breaks engineering teams. Not because the controls are hard to implement. They are already implemented. The problem is that nobody collected evidence while doing the work, nobody mapped the technical controls to the framework language, and now the team spends three weeks scrambling for screenshots and access logs instead of shipping product. **Compliance as code** solves this by making your CI/CD pipeline, your infrastructure, and your monitoring system generate audit evidence automatically as a side effect of doing normal engineering work. By the time the auditor arrives, the evidence is already collected. This module covers three things: understanding what SOC 2, ISO 27001, and PCI-DSS actually require from an engineering team, automating compliance using OPA and Cloud Custodian, and running incident response from detection through to blameless postmortem. ---
### What SOC 2 actually asks your pipeline to prove **SOC 2** (System and Organization Controls 2) is an auditing framework developed by the AICPA. It is the standard most SaaS companies and cloud-native startups need when enterprise customers ask "how do you protect our data?" SOC 2 is built on five **Trust Services Criteria (TSC)**: * **Security** — are your systems protected from unauthorized access? This covers firewalls, MFA, encryption, vulnerability management, and access reviews. * **Availability** — do your systems meet your uptime commitments? This covers monitoring, incident response, backup, and disaster recovery. * **Processing Integrity** — does your system process data completely and accurately? This covers data validation, change management, and error handling. * **Confidentiality** — is sensitive data protected throughout its lifecycle? This covers encryption, access controls, and data classification. * **Privacy** — do you handle personal data responsibly? This covers data collection policies, consent, GDPR alignment, and retention. Most startups only need the Security criterion. The others are optional unless your customers specifically require them. **Type I vs Type II** is the most important distinction: * **Type I** — a point-in-time audit. The auditor checks whether your controls are designed correctly as of a specific date. Takes 2-4 weeks to prepare. * **Type II** — an audit over a period of time, usually 6-12 months. The auditor tests whether your controls operated effectively throughout that period. This is what enterprise customers actually require. > 📌 **Remember:** SOC 2 Type II requires an observation period. You cannot get a Type II report on controls you implemented last week. Start collecting evidence from day one of implementing a control — not after an audit is scheduled. **What evidence looks like for a DevOps team:** The table below maps SOC 2 Security controls to the engineering artifacts that satisfy them: | SOC 2 Control | What the Auditor Checks | Engineering Evidence | |:---|:---|:---| | Logical access controls | Only authorized users access production | IAM access logs, PR review records | | Multi-factor authentication | MFA enabled for all production access | AWS CloudTrail MFA events | | Change management | All changes reviewed before deployment | GitHub PR history with approvals | | Vulnerability management | CVEs identified and remediated | Trivy scan reports, Snyk findings | | Incident response | Documented process for handling incidents | Incident tickets, postmortem docs | | Encryption | Data encrypted at rest and in transit | AWS Config rule results, TLS certificates | The key insight from engineers who have been through SOC 2 audits: **your CI/CD pipeline already produces most of this evidence.** PR approvals, pipeline run logs, scan reports, deployment history — these are all audit artifacts. The problem is they are not organized or retained with compliance in mind. ### What ISO 27001 asks for differently **ISO 27001** is an international standard for Information Security Management Systems (ISMS). Where SOC 2 is a report produced by an auditor for your customers, ISO 27001 is a certification your organization earns by building a systematic security management process. The 2022 version has 93 controls organized into four domains: * Organizational controls (37 controls) — policies, roles, supplier management * People controls (8 controls) — training, screening, disciplinary process * Physical controls (14 controls) — building security, equipment protection * Technological controls (34 controls) — access control, cryptography, development security The controls directly relevant to a DevSecOps team are in the technological domain. The most important ones for your pipeline: * **A.8.25 Secure Development Lifecycle** — security must be integrated into every phase of development, not added after * **A.8.26 Application Security Requirements** — security requirements must be defined before code is written * **A.8.27 Secure System Architecture** — systems must be designed with security as a default * **A.8.28 Secure Coding** — developers must follow secure coding standards * **A.8.29 Security Testing in Development** — SAST, DAST, and penetration testing must happen before production * **A.8.31 Separation of Development, Test, and Production** — code must not be tested in production > 💡 **Tip:** ISO 27001 auditors are not trying to catch you out. They want to see that you have a systematic process for managing security risks. A documented, imperfect process is better than an undocumented perfect one. Write down what you do before the auditor arrives. **The critical difference from SOC 2:** ISO 27001 requires a **risk register** — a documented list of information security risks, their likelihood, their impact, and the controls you chose to address them. This is not optional. Without a risk register, you cannot get ISO 27001 certified. ### What PCI-DSS adds for payment systems **PCI-DSS v4.0** (Payment Card Industry Data Security Standard) applies to any organization that processes, stores, or transmits payment card data. If your application takes credit card payments, you are in scope. The standard has 12 requirement areas. The ones most relevant to a DevSecOps engineer: * **Requirement 2** — all systems must have secure configurations (no defaults, no unnecessary services) * **Requirement 3** — cardholder data must be protected at rest with encryption * **Requirement 6** — software must be developed securely with vulnerability management * **Requirement 7** — access to system components must be restricted by business need * **Requirement 10** — all access to system components must be logged and monitored * **Requirement 11** — security systems and processes must be tested regularly For Kubernetes workloads specifically, PCI-DSS v4.0 requires: * Pod Security Standards enforced (no privileged containers in CDE namespaces) * Network Policies restricting traffic between pods * RBAC scoped to least privilege * Image scanning with Trivy before deployment * Runtime monitoring with Falco for the cardholder data environment > ⚠️ **Security:** PCI-DSS scope creep is a real risk. Every system that can communicate with your cardholder data environment (CDE) is potentially in scope. Use network segmentation aggressively to keep the CDE boundary small. A smaller scope means fewer systems to audit and secure. ### How to map one control across multiple frameworks simultaneously The most time-efficient compliance approach is to implement a control once and map it to every framework that requires it. The flow looks like this: One technical control implemented | v Mapped to SOC 2 Security criterion | v Mapped to ISO 27001 Annex A control | v Mapped to PCI-DSS requirement | v Single evidence artifact satisfies all three For example, implementing Checkov in your CI pipeline to scan Terraform before apply: * SOC 2: satisfies change management and configuration controls * ISO 27001: satisfies A.8.25 (Secure Development Lifecycle) and A.8.29 (Security Testing) * PCI-DSS: satisfies Requirement 2 (secure system configurations) and Requirement 6 (vulnerability management) One tool, one pipeline stage, three frameworks covered. ---
### Why manual compliance does not scale At Razorpay or PhonePe, thousands of infrastructure changes happen every week. A developer creates an S3 bucket. An EC2 instance is launched. A security group is modified. A Lambda function is deployed. Manual compliance means a security team reviews these changes periodically — maybe weekly, maybe monthly. In the gap between reviews, non-compliant resources sit in your environment. When the auditor arrives, they find violations. You spend two weeks remediating instead of two hours. Compliance as code means every change is evaluated against your policies automatically. A non-compliant resource is either blocked before it deploys (shift-left enforcement) or flagged and remediated within minutes of being created (continuous monitoring). Developer writes Terraform | v Checkov scans in CI pipeline | v Non-compliant config blocked before apply | |--- if passes ---v terraform apply | v AWS Config evaluates running resource | v Non-compliant resource triggers remediation ### How OPA enforces policies in pipelines and Kubernetes **Open Policy Agent (OPA)** is a general-purpose policy engine. You write policies in **Rego** — a declarative language designed for expressing rules about structured data. OPA evaluates your Rego policies against any JSON input and returns a decision. > **Note:** Rego is a declarative language, not an imperative one. You describe what is not allowed, not a sequence of steps to check it. This takes some getting used to but makes policies very readable once you learn the pattern. A simple OPA policy blocking S3 buckets without encryption: ```rego ## policy/s3_encryption.rego ## Deny S3 bucket resources that do not have encryption configured package terraform.aws ## deny rule — if this evaluates to true, the resource is blocked deny[msg] { ## iterate over all aws_s3_bucket resources in the plan resource := input.resource.aws_s3_bucket[bucket_name] ## check if server_side_encryption_configuration is missing not resource.server_side_encryption_configuration ## message shown to the developer when blocked msg := sprintf( "S3 bucket '%s' must have server_side_encryption_configuration", [bucket_name] ) } ``` Running this with **Conftest** (the CLI tool that wraps OPA for CI/CD): ```bash ## Install conftest curl -L https://github.com/open-policy-agent/conftest/releases/download/v0.66.0/conftest_0.66.0_Linux_x86_64.tar.gz | tar xzf - ## Run against a Terraform plan terraform plan -out=plan.binary terraform show -json plan.binary > plan.json ## Evaluate policies against the plan conftest test plan.json --policy policy/ ## Expected output when a violation is found: ## FAIL - plan.json - terraform.aws - S3 bucket 'prod-data' must have server_side_encryption_configuration ## 1 test, 0 passed, 0 warnings, 1 failure, 0 exceptions ``` > **Note:** A non-zero exit code from Conftest fails your CI pipeline. This is exactly what you want — a misconfigured resource cannot be deployed without the developer fixing the policy violation first. For Kubernetes admission control, **OPA Gatekeeper** runs as a webhook inside your cluster. Every resource submitted to the API server is evaluated against your policies before it is created. ```yaml ## gatekeeper-constraint-template.yaml ## Defines a new policy type called K8sRequireNonRoot apiVersion: templates.gatekeeper.sh/v1 kind: ConstraintTemplate metadata: name: k8srequirenonroot spec: crd: spec: names: kind: K8sRequireNonRoot targets: - target: admission.k8s.gatekeeper.sh rego: | package k8srequirenonroot ## deny if the container is configured to run as root violation[{"msg": msg}] { container := input.review.object.spec.containers[_] not container.securityContext.runAsNonRoot msg := sprintf( "Container '%s' must set runAsNonRoot: true", [container.name] ) } ``` ```yaml ## gatekeeper-constraint.yaml ## Apply the policy to all pods in the production namespace apiVersion: constraints.gatekeeper.sh/v1beta1 kind: K8sRequireNonRoot metadata: name: require-non-root-production spec: match: namespaces: ["production"] kinds: - apiGroups: [""] kinds: ["Pod"] ``` ```bash ## Apply the template and constraint kubectl apply -f gatekeeper-constraint-template.yaml kubectl apply -f gatekeeper-constraint.yaml ## Test — try to create a pod running as root kubectl run test-root --image=nginx -n production ## Error from server: admission webhook denied the request: ## Container 'test-root' must set runAsNonRoot: true ``` ### How Cloud Custodian automates cloud governance and remediation **Cloud Custodian** is an open-source tool that continuously monitors your cloud resources against policies and can automatically remediate violations. Where Checkov and Conftest catch problems before deployment, Cloud Custodian catches problems in already-running resources. Cloud Custodian policies are written in YAML and describe: * Which resource type to check (S3 bucket, EC2 instance, RDS database) * Filters to narrow down which resources to check * Actions to take on non-compliant resources A policy that finds unencrypted S3 buckets and tags them for remediation: ```yaml ## custodian-s3-encryption.yaml policies: - name: s3-unencrypted-buckets ## check all S3 buckets resource: aws.s3 ## filter to only buckets without encryption filters: - type: bucket-encryption state: False ## actions: tag the bucket and notify the team actions: - type: tag tags: ## mark it so the infrastructure team knows to fix it compliance-violation: "unencrypted-bucket" violation-date: "{now:%Y-%m-%d}" - type: notify template: default subject: "Unencrypted S3 bucket found: {account}/{bucket}" to: - slack://devsecops-alerts transport: type: sqs queue: https://sqs.ap-south-1.amazonaws.com/123456789/custodian-alerts ``` ```bash ## Run the policy in dry-run mode first — see what would be affected custodian run --dryrun -s output/ custodian-s3-encryption.yaml ## Run for real custodian run -s output/ custodian-s3-encryption.yaml ## View results custodian report -s output/ custodian-s3-encryption.yaml ``` A policy with automatic remediation — disable public access on non-compliant S3 buckets: ```yaml policies: - name: s3-block-public-access resource: aws.s3 filters: ## find buckets where public access block is not fully enabled - type: value key: "PublicAccessBlockConfiguration.BlockPublicAcls" value: false actions: ## automatically enable public access block - type: set-public-block BlockPublicAcls: true BlockPublicPolicy: true IgnorePublicAcls: true RestrictPublicBuckets: true ``` > ⚠️ **Security:** Always test auto-remediation policies in dry-run mode first in a non-production account. A policy that accidentally modifies the wrong resources can cause outages. Validate the filter logic thoroughly before enabling automatic actions. ### How AWS Config provides continuous compliance monitoring **AWS Config** records the configuration of every AWS resource and evaluates it against rules. Unlike Cloud Custodian which you run on a schedule, AWS Config runs continuously — every time a resource is created or modified, it is re-evaluated. ```bash ## Enable AWS Config with Terraform ``` ```hcl ## aws-config.tf ## Enable Config recorder to track all resources resource "aws_config_configuration_recorder" "main" { name = "devsecops-config-recorder" role_arn = aws_iam_role.config.arn recording_group { all_supported = true ## record all resource types include_global_resource_types = true ## include IAM, Route53, etc } } ## CIS Benchmark rule: S3 buckets must not allow public read resource "aws_config_config_rule" "s3_no_public_read" { name = "s3-bucket-public-read-prohibited" source { owner = "AWS" source_identifier = "S3_BUCKET_PUBLIC_READ_PROHIBITED" } depends_on = [aws_config_configuration_recorder.main] } ## CIS Benchmark rule: EBS volumes must be encrypted resource "aws_config_config_rule" "ebs_encryption" { name = "encrypted-volumes" source { owner = "AWS" source_identifier = "ENCRYPTED_VOLUMES" } } ## CIS Benchmark rule: CloudTrail must be enabled resource "aws_config_config_rule" "cloudtrail_enabled" { name = "cloudtrail-enabled" source { owner = "AWS" source_identifier = "CLOUD_TRAIL_ENABLED" } } ## Auto-remediation: disable public access on non-compliant S3 buckets resource "aws_config_remediation_configuration" "s3_public_read" { config_rule_name = aws_config_config_rule.s3_no_public_read.name target_type = "SSM_DOCUMENT" target_id = "AWS-DisableS3BucketPublicReadWrite" parameter { name = "S3BucketName" resource_value = "RESOURCE_ID" ## AWS Config substitutes the actual bucket name } automatic = true ## fix automatically without human approval maximum_automatic_attempts = 3 retry_attempt_seconds = 60 } ``` AWS Security Hub aggregates findings from Config, GuardDuty, Inspector, and Macie into one dashboard. Enable compliance standards to get a compliance score against CIS, PCI-DSS, and AWS best practices: ```bash ## Enable Security Hub and compliance standards via AWS CLI aws securityhub enable-security-hub --enable-default-standards ## Enable PCI-DSS standard aws securityhub batch-enable-standards \ --standards-subscription-requests \ '[{"StandardsArn":"arn:aws:securityhub:ap-south-1::standards/pci-dss/v/4.0.1"}]' ## Get compliance score summary aws securityhub get-findings \ --filters '{"ComplianceStatus":[{"Value":"FAILED","Comparison":"EQUALS"}]}' \ --query 'Findings[*].{Title:Title,Severity:Severity.Label,Resource:Resources[0].Id}' \ --output table ``` ---
### What evidence auditors actually check The most common reason compliance audits fail is not missing controls. It is missing evidence. An auditor cannot take your word for it. They need artifacts they can inspect — logs, reports, records — that prove the control operated during the audit period. Evidence falls into three categories: * **Automated artifacts** — scan reports, pipeline logs, access logs, configuration snapshots. These are generated without human effort and are the most reliable evidence. * **Documented processes** — runbooks, policies, procedures. These tell the auditor how you do something; the automated artifacts prove you actually did it. * **Human records** — meeting notes, approval emails, training completion records. These satisfy the people and organizational controls. For a DevSecOps team, the automated artifacts are the most valuable and the most underused. ### How to structure an evidence pipeline The goal is to have evidence generated and stored automatically every time a control runs. Not after an audit is scheduled. Not once a quarter. Every time. CI pipeline runs | v Trivy scan report generated | v Report stored in S3 with timestamp | v AWS Config records resource state | v CloudTrail logs every API call | v GitHub retains PR approval history | v Auditor requests evidence | v Pull from S3, Config, CloudTrail, GitHub Evidence is already there Configuring S3 for evidence retention with Terraform: ```hcl ## evidence-bucket.tf resource "aws_s3_bucket" "compliance_evidence" { bucket = "razorpay-devsecops-compliance-evidence-prod" } ## Enable versioning — never lose an evidence artifact resource "aws_s3_bucket_versioning" "evidence" { bucket = aws_s3_bucket.compliance_evidence.id versioning_configuration { status = "Enabled" } } ## Lifecycle policy — keep evidence for 3 years (SOC 2 requires 1 year minimum) resource "aws_s3_bucket_lifecycle_configuration" "evidence_retention" { bucket = aws_s3_bucket.compliance_evidence.id rule { id = "retain-compliance-evidence" status = "Enabled" expiration { days = 1095 ## 3 years } } } ## Block all public access — compliance evidence must never be public resource "aws_s3_bucket_public_access_block" "evidence" { bucket = aws_s3_bucket.compliance_evidence.id block_public_acls = true block_public_policy = true ignore_public_acls = true restrict_public_buckets = true } ``` In your GitHub Actions pipeline, store scan reports to the evidence bucket automatically: ```yaml ## .github/workflows/devsecops-pipeline.yml - name: Run Trivy scan run: | trivy image \ --format json \ --output trivy-results.json \ myapp:${{ github.sha }} - name: Store scan evidence in S3 run: | ## Upload with a structured key for easy retrieval during audit aws s3 cp trivy-results.json \ s3://razorpay-devsecops-compliance-evidence-prod/trivy-scans/$(date +%Y/%m/%d)/trivy-${{ github.sha }}.json ## Also upload the pipeline run metadata echo '{ "pipeline_run": "${{ github.run_id }}", "commit": "${{ github.sha }}", "image": "myapp:${{ github.sha }}", "scan_time": "'$(date -u +%Y-%m-%dT%H:%M:%SZ)'" }' | aws s3 cp - \ s3://razorpay-devsecops-compliance-evidence-prod/pipeline-runs/${{ github.run_id }}/metadata.json ``` > 💡 **Tip:** Use consistent S3 key prefixes structured by year/month/day and control type. When an auditor asks for "all vulnerability scan reports from January to March," you can run a single `aws s3 ls` command to produce the list. Unstructured evidence is almost as bad as missing evidence. ---
### Understanding the NIST SP 800-61 Rev 3 lifecycle **NIST SP 800-61 Rev 3** (released April 2025, replacing Rev 2) aligns incident response with the NIST Cybersecurity Framework 2.0. The new model has three active IR phases at the top level: Detect (monitoring finds the problem) | v Respond (contain, eradicate, communicate) | v Recover (restore service, collect lessons) | v Improve (feed lessons back into all functions) Below the active IR phases are the preparation activities — Govern, Identify, and Protect — which are broader risk management functions that also support incident response. You cannot have good incident response without preparation, but preparation itself is not part of responding to an incident. > **Note:** NIST Rev 2 used a four-phase lifecycle: Preparation, Detection and Analysis, Containment/Eradication/Recovery, and Post-Incident Activity. Rev 3 withdrew this model in April 2025. If your IR plan still references the Rev 2 phases, it needs updating. Both models cover the same ground — the new model just integrates more naturally with broader security governance. ### Building an incident response plan that engineers will actually use Most IR plans fail not because they are wrong but because they are written in language that security teams understand and engineers ignore. The plan needs to answer one question: "it is 2 AM, production is down, what do I do right now?" A practical IR plan has three parts: **Part 1 — Severity classification:** | Severity | Definition | Response Time | Escalation | |:---|:---|:---|:---| | P1 Critical | Data breach, complete outage, active attack | Immediate - wake everyone | CTO, legal within 15 min | | P2 High | Partial outage, credential compromise, ransomware detected | 15 minutes | Engineering manager, security team | | P3 Medium | Single service degraded, suspicious activity | 1 hour | On-call engineer | | P4 Low | Non-critical finding, no active exploitation | Next business day | Standard ticket | **Part 2 — Containment playbooks per incident type:** A playbook for a compromised AWS credential: ```bash ## PLAYBOOK: Compromised AWS Access Key ## Severity: P2 High ## Owner: Security team + affected service owner ## Step 1 — Disable the key immediately (do this first, before anything else) aws iam update-access-key \ --access-key-id AKIAIOSFODNN7EXAMPLE \ --status Inactive \ --user-name affected-service-account ## Step 2 — Investigate what the key was used for ## Check the last 24 hours of CloudTrail activity for this key aws cloudtrail lookup-events \ --lookup-attributes AttributeKey=AccessKeyId,AttributeValue=AKIAIOSFODNN7EXAMPLE \ --start-time $(date -u -d "24 hours ago" +%Y-%m-%dT%H:%M:%SZ) \ --query 'Events[*].{Time:EventTime,Event:EventName,Source:EventSource}' \ --output table ## Step 3 — Look specifically for signs of lateral movement ## New IAM users or access keys created by the attacker aws iam list-users \ --query 'Users[?CreateDate>=`'$(date -u -d "24 hours ago" +%Y-%m-%dT%H:%M:%S)'`]' ## EC2 instances launched (cryptomining) aws ec2 describe-instances \ --filters "Name=launch-time,Values=$(date -u -d '24 hours ago' +%Y-%m-%dT%H:%M:%S)*" \ --query 'Reservations[*].Instances[*].{ID:InstanceId,Type:InstanceType,Launch:LaunchTime}' ## Step 4 — Generate a new key and update all systems aws iam create-access-key --user-name affected-service-account ## Update: Kubernetes secrets, CI environment variables, application config ## Step 5 — Delete the old key permanently aws iam delete-access-key \ --access-key-id AKIAIOSFODNN7EXAMPLE \ --user-name affected-service-account ``` **Part 3 — Communication templates:** The first update to stakeholders within 15 minutes of a P1: ```text INCIDENT: [P1] Potential data exposure in payment service Time detected: 14:32 IST Status: Investigating Impact: Payment processing may be affected for Swiggy merchant accounts What we know: Unusual API calls detected from service account prod-payment-api Actions taken: Service account disabled, investigation in progress Next update: 14:47 IST Owner: rahul@company.com ``` > 📌 **Remember:** The first communication to stakeholders should go out within 15 minutes of declaring a P1 incident, even if you know nothing yet. "We are investigating and will update in 15 minutes" is infinitely better than silence. Silence is what causes executives to escalate and interrupt the engineers who are trying to fix the problem. ### Containment before eradication — the sequence that most teams get wrong The single most common IR mistake is jumping to recovery before eradication is complete. The sequence must be: 1. CONTAIN (stop the bleeding — isolate compromised system, revoke credentials, block attacker path) | v 2. ERADICATE (remove the threat — delete malware, close the vulnerability, rotate all affected credentials) | v 3. RECOVER (restore service — bring systems back up, verify clean state, restore from backup if needed) If you restore service before eradicating the threat, the attacker still has access. You have recovered a compromised system. This happens frequently when business pressure to restore service overrides the technical steps. For a Kubernetes incident — containing a compromised pod: ```bash ## CONTAIN: Isolate the compromised pod immediately ## Apply a NetworkPolicy that blocks all ingress and egress kubectl apply -f - <<EOF apiVersion: networking.k8s.io/v1 kind: NetworkPolicy metadata: name: isolate-compromised-pod namespace: production spec: podSelector: matchLabels: ## target only the compromised pod by its specific label app: payment-service pod-name: payment-service-7d8f9b-x4k2p ## empty ingress/egress means deny everything policyTypes: - Ingress - Egress EOF ## CONTAIN: Prevent the pod from being rescheduled kubectl cordon <node-name> ## ERADICATE: Capture forensic snapshot before killing the pod ## Save logs kubectl logs payment-service-7d8f9b-x4k2p -n production > incident-pod-logs.txt ## Save pod description and events kubectl describe pod payment-service-7d8f9b-x4k2p -n production > incident-pod-describe.txt ## ERADICATE: Now delete the compromised pod kubectl delete pod payment-service-7d8f9b-x4k2p -n production ## RECOVER: Deploy a clean pod from the known-good image kubectl rollout restart deployment/payment-service -n production ## Verify the new pod is healthy kubectl rollout status deployment/payment-service -n production ``` ### Writing a blameless postmortem that produces real improvements A postmortem is not a way to find out who broke production. It is a structured process for understanding why a complex system failed and how to make it more resilient. **Blameless** means the postmortem focuses on systems, processes, and conditions — not on individual mistakes. Engineers who fear blame hide information. Engineers who feel safe share everything. You need everything to understand what actually happened. A good postmortem has five sections: **1. Timeline** — a factual, minute-by-minute record of what happened: ```text 14:28 - Automated alert fires: payment service P99 latency > 5s 14:31 - On-call engineer Ravi acknowledges alert 14:35 - Ravi identifies database connection pool at 100% utilization 14:41 - Ravi increases connection pool limit from 50 to 100 14:43 - Latency begins to improve 14:51 - Latency returns to baseline, incident resolved 14:52 - P2 incident closed, postmortem scheduled ``` **2. Root cause** — what was the actual technical cause (not "human error"): ```text Root cause: A new query introduced in the v2.3.1 release scans the full transactions table without using an index. Under normal load this adds 200ms per query. During the Diwali sale traffic spike (3x normal volume), the query saturated all available connections waiting for slow scans. ``` **3. Contributing factors** — conditions that made the incident worse: * No index on the `transactions.merchant_id` column (architectural gap) * No alert on database query time, only on connection pool utilization (observability gap) * Connection pool size was never re-evaluated after the traffic pattern changed 6 months ago (process gap) **4. Impact** — measurable business impact: * 23 minutes of elevated latency (P99 > 5s) * 0 transactions failed (connection pool exhausted but requests queued successfully) * 0 data loss **5. Action items** — specific, assigned, time-bound: | Action | Owner | Due Date | |:---|:---|:---| | Add index on transactions.merchant\_id | Priya (backend) | Within 24 hours | | Add Prometheus alert for slow query time | Ravi (SRE) | Within 48 hours | | Review connection pool size quarterly | Platform team | Recurring quarterly | | Add slow query check to PR review checklist | Engineering manager | Within 1 week | > 🔴 **Common Mistake:** Postmortem action items that say "improve monitoring" or "be more careful" are not action items. Every action item must name a specific person, a specific change, and a specific date. "Improve monitoring" assigned to "the team" with no due date will never happen. ---
1. Create a new directory and initialize a Terraform project with a deliberately misconfigured S3 bucket: ```bash mkdir compliance-lab && cd compliance-lab cat > main.tf << 'EOF' resource "aws_s3_bucket" "data" { bucket = "hotstar-compliance-lab-bucket" } ## Intentionally missing: server_side_encryption_configuration ## Intentionally missing: public access block EOF ``` 2. Install Conftest and create an OPA policy for S3 encryption: ```bash ## Download Conftest binary curl -L https://github.com/open-policy-agent/conftest/releases/download/v0.66.0/conftest_0.66.0_Linux_x86_64.tar.gz \ | tar xzf - && sudo mv conftest /usr/local/bin/ ## Create policy directory and write the S3 encryption policy mkdir policy cat > policy/s3_security.rego << 'EOF' package terraform.aws ## Deny S3 buckets without encryption deny[msg] { resource := input.resource.aws_s3_bucket[bucket_name] not resource.server_side_encryption_configuration msg := sprintf("S3 bucket '%s' must have encryption configured", [bucket_name]) } ## Deny S3 buckets without public access block deny[msg] { resource := input.resource.aws_s3_bucket[bucket_name] not input.resource.aws_s3_bucket_public_access_block msg := sprintf("S3 bucket '%s' must have public access block configured", [bucket_name]) } EOF ``` 3. Generate a Terraform plan and run Conftest against it: ```bash ## Initialize and plan terraform init terraform plan -out=plan.binary terraform show -json plan.binary > plan.json ## Run Conftest — this should fail with 2 violations conftest test plan.json --policy policy/ ## Expected output: ## FAIL - plan.json - terraform.aws - S3 bucket 'hotstar-compliance-lab-bucket' must have encryption configured ## FAIL - plan.json - terraform.aws - S3 bucket 'hotstar-compliance-lab-bucket' must have public access block configured ## 2 tests, 0 passed, 0 warnings, 2 failures ``` 4. Fix the Terraform code to pass both policies: ```bash cat > main.tf << 'EOF' resource "aws_s3_bucket" "data" { bucket = "hotstar-compliance-lab-bucket" } ## Fix 1: Add encryption resource "aws_s3_bucket_server_side_encryption_configuration" "data" { bucket = aws_s3_bucket.data.id rule { apply_server_side_encryption_by_default { sse_algorithm = "AES256" } } } ## Fix 2: Block public access resource "aws_s3_bucket_public_access_block" "data" { bucket = aws_s3_bucket.data.id block_public_acls = true block_public_policy = true ignore_public_acls = true restrict_public_buckets = true } EOF ## Re-generate plan and re-run Conftest terraform plan -out=plan.binary terraform show -json plan.binary > plan.json conftest test plan.json --policy policy/ ## Expected output: ## 2 tests, 2 passed, 0 warnings, 0 failures ``` 5. Install Cloud Custodian and run a dry-run compliance check against your AWS account: ```bash ## Install Cloud Custodian pip install c7n --break-system-packages ## Create a policy to find unencrypted S3 buckets in your account cat > custodian-check.yaml << 'EOF' policies: - name: find-unencrypted-buckets resource: aws.s3 filters: - type: bucket-encryption state: False actions: - type: tag tags: compliance-violation: unencrypted-bucket EOF ## Run in dry-run mode first — never run custodian without dry-run first custodian run \ --dryrun \ --region ap-south-1 \ --output-dir custodian-output \ custodian-check.yaml ## View which buckets would be affected cat custodian-output/find-unencrypted-buckets/resources.json | python3 -m json.tool ``` 6. Simulate an incident and write a postmortem. Create a deliberately broken deployment: ```bash ## Simulate: deploy a pod running as root (a security violation) cat > bad-pod.yaml << 'EOF' apiVersion: v1 kind: Pod metadata: name: incident-simulation namespace: default spec: containers: - name: app image: nginx:latest securityContext: runAsUser: 0 ## running as root — this is the violation EOF kubectl apply -f bad-pod.yaml ## Detect: check running pods for root containers kubectl get pods -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.spec.containers[*].securityContext.runAsUser}{"\n"}{end}' ## Contain: apply a network policy to isolate the pod kubectl apply -f - <<EOF apiVersion: networking.k8s.io/v1 kind: NetworkPolicy metadata: name: isolate-bad-pod spec: podSelector: matchLabels: app: incident-simulation policyTypes: - Ingress - Egress EOF ## Eradicate: collect logs then delete kubectl logs incident-simulation > incident-logs.txt kubectl delete pod incident-simulation ## Write a postmortem document (template) cat > postmortem-$(date +%Y%m%d).md << 'EOF' # Postmortem: Root Container Deployed to Production Date: $(date +%Y-%m-%d) Severity: P3 Medium Duration: [fill in minutes from detect to resolve] ## Timeline [fill in minute-by-minute] ## Root Cause [what was the actual technical cause] ## Contributing Factors [what made it possible for this to happen] ## Impact [what was affected, for how long] ## Action Items | Action | Owner | Due Date | |--------|-------|----------| | Add OPA policy blocking root containers | [name] | [date] | | Add Falco rule alerting on root container spawn | [name] | [date] | EOF echo "Postmortem template created: postmortem-$(date +%Y%m%d).md" ``` ---
Your startup just won a contract with a large enterprise. Their procurement team sends a security questionnaire with 180...
What SOC 2 actually asks your pipeline to prove SOC 2 (System and Organization Controls 2) is an auditing framework deve...
Why manual compliance does not scale At Razorpay or PhonePe, thousands of infrastructure changes happen every week. A de...
What evidence auditors actually check The most common reason compliance audits fail is not missing controls. It is missi...
Understanding the NIST SP 800-61 Rev 3 lifecycle NIST SP 800-61 Rev 3 (released April 2025, replacing Rev 2) aligns inci...
Create a new directory and initialize a Terraform project with a deliberately misconfigured S3 bucket: Install Conftest ...
Tool What it does When to use conftest test plan.json Evaluate Terraform plan against OPA policies In CI before terrafor...
Engineers going through their first compliance audit almost always make the same set of mistakes. Understanding them bef...
Aligns directly with DevOps, Site Reliability (SRE), and Platform Engineering job descriptions.