DevSecOps project - Build a Fully Gated Secure CI/CD Pipeline | Engineering Modules | DevOps Network

Engineering ModuleADVANCED

DevSecOps project - Build a Fully Gated Secure CI/CD Pipeline

Build a production-grade secure pipeline where every push triggers SAST, SCA, secrets detection, container scanning, SBOM generation, and image signing before any code reaches production.

~210 Minutes

14 Topics

Hands-on Scenarios

What You'll Learn

The Incident That Started Everything

It is a Thursday afternoon at Razorpay. A junior engineer pushes a payment processing microservice to GitHub. The code looks clean. The PR passes code review. The pipeline runs. Deployment succeeds. Forty-eight hours later, AWS sends a security alert. An IAM key hardcoded in the repository has been used from an IP in Singapore to create hundreds of EC2 instances. The key has been live in the repo for six weeks, exposed in commit history even after someone noticed and deleted the file. The damage: $140,000 in unauthorized cloud spend, a 12-hour incident response, and a compliance audit. Here is what a proper DevSecOps pipeline would have done: caught the secret in under 4 seconds, blocked the push before it ever reached the remote, and sent a structured alert to Slack with the exact file and line number. This capstone builds that pipeline from scratch. By the end, one `git push` from a developer triggers seven automated security gates. Every gate either passes and lets the code move forward, or fails with a structured report explaining exactly what is wrong and how to fix it.

Understanding the Security Gates

### The seven gates and what each one catches A **security gate** is a mandatory check that blocks deployment if it finds a violation. Unlike optional linting, a failing gate stops the pipeline completely. No gate failure means no deployment. Here are the seven gates you are building: ┌─────────────────────────────────────────────────────────┐ │ git push │ └──────────────────────────┬──────────────────────────────┘ │ ▼ ┌──────────────────────────────────────────────────────────┐ │ Gate 1: Secrets Detection (Gitleaks) │ │ Catches: hardcoded API keys, passwords, tokens │ │ Time: ~4 seconds │ └──────────────────────────┬───────────────────────────────┘ │ pass ▼ ┌──────────────────────────────────────────────────────────┐ │ Gate 2: SAST — Static Analysis (Semgrep) │ │ Catches: SQL injection, path traversal, unsafe evals │ │ Time: ~45 seconds │ └──────────────────────────┬───────────────────────────────┘ │ pass ▼ ┌──────────────────────────────────────────────────────────┐ │ Gate 3: SCA — Dependency Scan (Trivy fs) │ │ Catches: CVEs in pip/npm/maven dependencies │ │ Time: ~30 seconds │ └──────────────────────────┬───────────────────────────────┘ │ pass ▼ ┌──────────────────────────────────────────────────────────┐ │ Gate 4: Container Image Build │ │ Builds the Docker image for scanning │ └──────────────────────────┬───────────────────────────────┘ │ pass ▼ ┌──────────────────────────────────────────────────────────┐ │ Gate 5: Container Image Scan (Trivy image) │ │ Catches: CVEs in the base OS, system packages │ │ Time: ~60 seconds │ └──────────────────────────┬───────────────────────────────┘ │ pass ▼ ┌──────────────────────────────────────────────────────────┐ │ Gate 6: SBOM Generation (Syft) │ │ Creates: full inventory of every package in the image │ │ Time: ~20 seconds │ └──────────────────────────┬───────────────────────────────┘ │ pass ▼ ┌──────────────────────────────────────────────────────────┐ │ Gate 7: Image Signing (Cosign) │ │ Signs: the image with a cryptographic key │ │ Time: ~10 seconds │ └──────────────────────────┬───────────────────────────────┘ │ pass ▼ ┌──────────────────────────────────────────────────────────┐ │ Deploy to production │ └──────────────────────────────────────────────────────────┘ ### Why each gate exists and what breaks without it **Gitleaks** solves the most expensive class of security incident: credential exposure. Keys committed to git are permanent — even if you delete the file, the key lives in git history. Gitleaks scans every commit diff, not just the current state of files. **Semgrep** (Static Application Security Testing or SAST) reads your source code without running it. It finds patterns that indicate vulnerabilities — `exec(user_input)` is dangerous regardless of what `user_input` contains. Semgrep uses community-maintained rule sets covering OWASP Top 10 for Python, Node.js, Java, and Go. **Trivy filesystem scan** (Software Composition Analysis or SCA) reads your `requirements.txt`, `package.json`, `go.mod`, or `pom.xml` and checks every dependency version against the CVE database. A library you have been using for two years might have a critical vulnerability disclosed last week. **Trivy image scan** goes further than the filesystem scan. After the Docker image is built, it scans the OS layer — the Ubuntu or Alpine base image packages, the system libraries. A Python app might be clean, but if it runs on an Ubuntu 20.04 base with an old glibc, that CVE is in your container. **Syft** generates a **Software Bill of Materials (SBOM)** — a complete inventory of every package, library, and OS component in the image with exact versions. This is required by government compliance frameworks (NIST SSDF, Executive Order 14028) and increasingly required by enterprise customers before they will use your software. **Cosign** signs the container image with a cryptographic key. When Kubernetes pulls the image, it can verify the signature before running it. This prevents supply chain attacks where someone replaces your image in the registry with a malicious version.

Setting Up the Project

### The application you are securing You are building a pipeline for a payment service — realistic enough to have actual vulnerabilities you will deliberately inject and then catch. ```bash ## Create the project directory mkdir razorpay-payment-service && cd razorpay-payment-service ## Set up Python virtual environment python3 -m venv venv && source venv/bin/activate ## Create the project structure mkdir -p src tests .github/workflows touch src/__init__.py touch src/app.py touch src/database.py touch src/auth.py touch requirements.txt touch Dockerfile touch .gitignore echo "✅ Project structure created" ``` ```python ## src/app.py ## A realistic Flask payment service with some intentional ## vulnerabilities you will catch and fix during this capstone. from flask import Flask, request, jsonify import sqlite3 import subprocess import os app = Flask(__name__) ## ⚠️ This file intentionally contains vulnerabilities for learning. ## The pipeline will catch them. You will fix them. @app.route("/payment/search") def search_payments(): """Search payments by merchant name.""" merchant = request.args.get("merchant", "") ## VULNERABILITY 1: SQL injection ## Semgrep will flag this as a SAST finding conn = sqlite3.connect("payments.db") cursor = conn.cursor() query = f"SELECT * FROM payments WHERE merchant = '{merchant}'" cursor.execute(query) results = cursor.fetchall() conn.close() return jsonify({"payments": results}) @app.route("/payment/export") def export_payments(): """Export payments to CSV file.""" filename = request.args.get("filename", "export.csv") ## VULNERABILITY 2: Path traversal ## Semgrep will flag this as a SAST finding with open(f"/exports/{filename}", "w") as f: f.write("payment_id,amount,merchant\n") return jsonify({"status": "exported", "file": filename}) @app.route("/health") def health(): return jsonify({"status": "ok", "service": "payment-service"}) if __name__ == "__main__": app.run(host="0.0.0.0", port=8080) ``` ```python ## src/auth.py ## Authentication module import hashlib import os ## VULNERABILITY 3: Hardcoded secret key ## Gitleaks will flag this immediately AWS_SECRET_KEY = "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY" DB_PASSWORD = "razorpay_prod_db_pass_2024" def hash_password(password: str) -> str: """Hash a password using SHA-256.""" ## SHA-256 is used here for demo — use bcrypt in production return hashlib.sha256(password.encode()).hexdigest() def verify_token(token: str) -> bool: """Verify a JWT token.""" ## Simplified for the capstone — use PyJWT in production return len(token) > 20 ``` ```text ## requirements.txt ## Note: these versions are intentionally outdated ## to trigger SCA findings in Trivy Flask==2.0.1 requests==2.25.0 cryptography==3.4.6 PyYAML==5.3.1 Pillow==8.1.0 ``` > 🔴 **Common Mistake:** Using `latest` as versions in requirements.txt makes Trivy unable to check for CVEs because it cannot identify the exact version. Always pin exact versions, even if old ones — then the scanner can tell you which ones are vulnerable. ```dockerfile ## Dockerfile ## Uses an intentionally older base image to trigger image scan findings ## VULNERABILITY 4: Old base image with known CVEs FROM python:3.9-slim-buster WORKDIR /app ## Copy requirements first for Docker layer caching COPY requirements.txt . ## Install dependencies RUN pip install --no-cache-dir -r requirements.txt ## Copy application code COPY src/ ./src/ ## Run as root (bad practice — the image scan will warn about this) EXPOSE 8080 CMD ["python", "src/app.py"] ``` ### Initializing git and the first commit ```bash ## Add a proper .gitignore before committing anything cat > .gitignore << 'EOF' venv/ __pycache__/ *.pyc *.pyo .env *.db *.log .pytest_cache/ EOF ## Initialize git git init git add . ## This first commit will fail when you add Gitleaks ## because src/auth.py contains hardcoded secrets ## That is intentional — you will fix it after seeing the gate in action git commit -m "Initial payment service setup" ```

Building Gate 1 — Secrets Detection with Gitleaks

### Installing and running Gitleaks locally **Gitleaks** is an open-source tool that scans git repositories for secrets using a pattern library of over 150 secret types. It checks file contents, commit history, and git blob diffs. ```bash ## Install Gitleaks (Linux/WSL) wget https://github.com/gitleaks/gitleaks/releases/download/v8.18.4/gitleaks_8.18.4_linux_x64.tar.gz tar xzf gitleaks_8.18.4_linux_x64.tar.gz sudo mv gitleaks /usr/local/bin/ gitleaks version ## Install on macOS brew install gitleaks ``` ```bash ## Run Gitleaks against the entire repository gitleaks detect --source . --verbose ## Expected output when secrets exist: ## ○ ## │╲ ## │ ○ ## ○ ░ ## ░ gitleaks ## ## Finding: AWS Secret Access Key ## Secret: wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY ## RuleID: aws-secret-access-key ## Entropy: 4.55 ## File: src/auth.py ## Line: 9 ## Commit: a3f2b1c ## Author: dev@razorpay.com ## Date: 2024-01-15T10:23:11Z ## Fingerprint: a3f2b1c:src/auth.py:aws-secret-access-key:9 ## ## 2 findings found. ## exit code 1 ``` > **Note:** The `exit code 1` is intentional and critical. When Gitleaks returns exit code 1, the GitHub Actions step fails and the pipeline stops. No code moves forward until secrets are removed from the codebase and its history. ### Fixing the secrets finding ```bash ## NEVER just delete the file containing the secret. ## The commit history still has it. You need environment variables. ## Fix src/auth.py — remove hardcoded credentials cat > src/auth.py << 'EOF' import hashlib import os ## ✅ Read from environment variables — never hardcode credentials AWS_SECRET_KEY = os.environ.get("AWS_SECRET_KEY") DB_PASSWORD = os.environ.get("DB_PASSWORD") if not AWS_SECRET_KEY: raise EnvironmentError("AWS_SECRET_KEY environment variable not set") def hash_password(password: str) -> str: """Hash a password using SHA-256.""" return hashlib.sha256(password.encode()).hexdigest() def verify_token(token: str) -> bool: """Verify a JWT token.""" return len(token) > 20 EOF ``` ```bash ## Create a .gitleaks.toml config to tune false positives cat > .gitleaks.toml << 'EOF' title = "Razorpay Payment Service Gitleaks Config" [allowlist] description = "Allowlisted patterns" ## Ignore example values in documentation regexes = [ '''EXAMPLE''', '''REPLACE_ME''', ] ## Ignore test files that contain fake credentials paths = [ '''tests/fixtures/''', ] EOF ``` ```bash ## Verify Gitleaks passes after the fix gitleaks detect --source . --verbose ## Expected: "No leaks found." ## exit code 0 echo "✅ Gate 1 passes" ```

Building Gate 2 — SAST with Semgrep

### Understanding what SAST finds **SAST (Static Application Security Testing)** analyzes source code for patterns that indicate vulnerabilities without running the code. It is pattern matching at a sophisticated level — not just looking for dangerous function names, but understanding the data flow from user input to dangerous operations. Semgrep uses a rule language that lets security engineers write custom rules in YAML. The community maintains thousands of rules covering OWASP Top 10 for every major language. ```bash ## Install Semgrep pip install semgrep ## Run Semgrep with the Python security ruleset semgrep --config=p/python src/ ## Expected output for our app.py: ## Scanning 3 files... ## ## src/app.py ## severity:error rule:python.lang.security.audit.formatted-sql-query ## 19| query = f"SELECT * FROM payments WHERE merchant = '{merchant}'" ## Detected SQL statement that is tainted by `request` object. ## Avoid string concatenation or formatting in SQL queries. ## [CWE-89: SQL Injection] ## ## src/app.py ## severity:warning rule:python.lang.security.audit.path-traversal.open ## 30| with open(f"/exports/{filename}", "w") as f: ## Found user-controlled `open()` call. ## [CWE-22: Path Traversal] ## ## Findings: 2 errors, 1 warning ## exit code 1 ``` ### Fixing the SAST findings ```python ## src/app.py — fixed version with all vulnerabilities remediated from flask import Flask, request, jsonify import sqlite3 import os import re app = Flask(__name__) ## ALLOWED_EXPORT_DIR constrains where files can be written ## Prevents path traversal by rejecting any filename with ../ ALLOWED_EXPORT_DIR = "/exports" ALLOWED_FILENAME_PATTERN = re.compile(r'^[a-zA-Z0-9_\-]+\.csv$') @app.route("/payment/search") def search_payments(): """Search payments by merchant name using parameterised query.""" merchant = request.args.get("merchant", "") ## ✅ FIXED: Parameterised query prevents SQL injection ## The ? placeholder means sqlite3 escapes the value automatically conn = sqlite3.connect("payments.db") cursor = conn.cursor() cursor.execute("SELECT * FROM payments WHERE merchant = ?", (merchant,)) results = cursor.fetchall() conn.close() return jsonify({"payments": results}) @app.route("/payment/export") def export_payments(): """Export payments to CSV with safe filename validation.""" filename = request.args.get("filename", "export.csv") ## ✅ FIXED: Validate filename with allowlist pattern before use if not ALLOWED_FILENAME_PATTERN.match(filename): return jsonify({"error": "Invalid filename format"}), 400 safe_path = os.path.join(ALLOWED_EXPORT_DIR, filename) ## ✅ Double-check resolved path stays within allowed directory if not os.path.realpath(safe_path).startswith(ALLOWED_EXPORT_DIR): return jsonify({"error": "Access denied"}), 403 with open(safe_path, "w") as f: f.write("payment_id,amount,merchant\n") return jsonify({"status": "exported", "file": filename}) @app.route("/health") def health(): return jsonify({"status": "ok", "service": "payment-service"}) if __name__ == "__main__": app.run(host="0.0.0.0", port=8080) ``` ```bash ## Re-run Semgrep to confirm zero findings semgrep --config=p/python src/ ## Expected: "No findings." ## exit code 0 echo "✅ Gate 2 passes" ```

Building Gate 3 — SCA with Trivy

### Scanning dependencies for CVEs **SCA (Software Composition Analysis)** checks your third-party dependencies against public vulnerability databases. **CVE (Common Vulnerabilities and Exposures)** is a list maintained by NIST — every known security flaw in public software gets a CVE ID, a severity score (CVSS), and a description of the impact. **Trivy** is an open-source scanner from Aqua Security. It checks packages from pip, npm, go modules, maven, gradle, and more against multiple CVE databases including NVD, GitHub Security Advisories, and OS vendor advisories. ```bash ## Install Trivy wget -qO - https://aquasecurity.github.io/trivy-repo/deb/public.key | sudo apt-key add - echo "deb https://aquasecurity.github.io/trivy-repo/deb generic main" \ | sudo tee /etc/apt/sources.list.d/trivy.list sudo apt update && sudo apt install trivy -y ## Scan the filesystem (checks requirements.txt) trivy fs --severity HIGH,CRITICAL --exit-code 1 . ``` ```text ## Expected output — our old dependencies have CVEs: ## 2024-01-15T10:30:00.000Z INFO Detected OS: unknown ## 2024-01-15T10:30:01.000Z INFO Scanning files... ## ## requirements.txt (pip) ## Total: 4 (HIGH: 2, CRITICAL: 2) ## ## ┌─────────────────┬──────────────────┬──────────┬────────────┬──────────────┐ ## │ Library │ Vulnerability │ Severity │ Installed │ Fixed In │ ## ├─────────────────┼──────────────────┼──────────┼────────────┼──────────────┤ ## │ cryptography │ CVE-2023-23931 │ CRITICAL │ 3.4.6 │ 39.0.1 │ ## │ requests │ CVE-2023-32681 │ HIGH │ 2.25.0 │ 2.31.0 │ ## │ PyYAML │ CVE-2022-1471 │ CRITICAL │ 5.3.1 │ 6.0 │ ## │ Pillow │ CVE-2023-44271 │ HIGH │ 8.1.0 │ 10.0.0 │ ## └─────────────────┴──────────────────┴──────────┴────────────┴──────────────┘ ``` ```text ## requirements.txt — FIXED with patched versions Flask==3.0.0 requests==2.31.0 cryptography==41.0.5 PyYAML==6.0.1 Pillow==10.1.0 ``` ```bash ## Re-run after updating requirements.txt trivy fs --severity HIGH,CRITICAL --exit-code 1 . ## Expected: "Total: 0 (HIGH: 0, CRITICAL: 0)" echo "✅ Gate 3 passes" ```

DevSecOps project - Build a Fully Gated Secure CI/CD Pipeline

What You'll Learn

The Incident That Started Everything

Understanding the Security Gates

Setting Up the Project

Building Gate 1 — Secrets Detection with Gitleaks

Building Gate 2 — SAST with Semgrep

Building Gate 3 — SCA with Trivy

Skills You'll Master

Curriculum Index

The Incident That Started Everything

Understanding the Security Gates

Setting Up the Project

Building Gate 1 — Secrets Detection with Gitleaks

Building Gate 2 — SAST with Semgrep

Building Gate 3 — SCA with Trivy

Building Gates 4 and 5 — Container Image Build and Scan

Building Gate 6 — SBOM Generation with Syft

Building Gate 7 — Image Signing with Cosign

Writing the Full GitHub Actions Pipeline

Running the Complete Pipeline Locally

Production Checklist

Common Production Mistakes

Quick Reference

Career Impact

Troubleshooting & Lab Guides

How to Troubleshoot a Linux Production Server: A Systematic Approach

Next Modules

S3 to FSx for OpenZFS

Senior DevOps Engineer Interview Prep

DevSecOps Basics

Related Guides

Shift-Left Security: Adding SBOM and Supply Chain Scanning to Your CI/CD Pipeline

GitHub Actions vs GitLab CI vs Jenkins: Which CI/CD Tool in 2025?