DevSecOps project 2 - Harden a Kubernetes Cluster End to End

Lock down a production Kubernetes cluster using RBAC, Pod Security Admission, NetworkPolicies, OPA Gatekeeper, Falco runtime detection, and automated compliance scanning.

Related Concepts & TermsRBAC

Domains & Technologies

Domains

CAPSTONEDEVSECOPSRBACFALCOOPA-GATEKEEPERNETWORK-POLICY

Technologies

KUBERNETES

Blueprint Walkthrough

The Breach That Happened Because of Defaults

A fintech startup running on Kubernetes had been in production for eight months. Their application was secure. Their code had no vulnerabilities. Their team was competent.

Then a developer accidentally pushed a pod manifest that had hostPID: true set — a leftover from debugging. The pod started successfully. An attacker who found a remote code execution vulnerability in the application could now escape the container and see every process running on the Kubernetes node — including processes from other customers' workloads on the same node.

The vulnerability was not in the code. It was in a missing policy. Nobody had told Kubernetes "never allow hostPID: true in this cluster." Kubernetes does exactly what you tell it to do. If you do not tell it what to refuse, it refuses nothing.

This capstone hardens a Kubernetes cluster using six layers of security controls. When you finish, a developer cannot accidentally — or deliberately — deploy a privileged container, access another service they should not, escalate their own permissions, or run a cryptocurrency miner without an alert firing within 60 seconds.

The six layers:

◈ DIAGRAM

┌─────────────────────────────────────────────────────────────┐
│  Layer 1: RBAC — Who can do what in the cluster             │
│  Layer 2: Pod Security Admission — What pods can run        │
│  Layer 3: NetworkPolicies — What can talk to what           │
│  Layer 4: OPA Gatekeeper — Custom admission policies        │
│  Layer 5: Falco — Runtime threat detection                  │
│  Layer 6: kube-bench — Automated CIS compliance scanning    │
└─────────────────────────────────────────────────────────────┘

Understanding the Attack Surfaces

What Kubernetes exposes by default

Kubernetes is secure by design in some areas and dangerously permissive in others. Understanding what is locked by default and what you need to lock yourself is the foundation of cluster hardening.

Locked by default:

etcd encryption at rest (if configured during cluster bootstrap)
API server TLS
Service account token mounting (can be disabled)

Open by default — you must close these:

All pods in a namespace can reach all other pods in the cluster on any port
A pod can request root (UID 0) inside the container
A pod can mount the host filesystem
Any authenticated user who can create pods can potentially escalate to cluster-admin
No runtime monitoring — if a container starts a shell or writes a binary, nothing alerts

Bash

## Set up a local cluster for this capstone
## Use kind (Kubernetes IN Docker) — it runs a full K8s cluster locally
 
## Install kind
curl -Lo ./kind https://kind.sigs.k8s.io/dl/v0.22.0/kind-linux-amd64
chmod +x ./kind && sudo mv ./kind /usr/local/bin/kind
 
## Install kubectl
curl -LO "https://dl.k8s.io/release/$(curl -L -s \
  https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl"
chmod +x kubectl && sudo mv kubectl /usr/local/bin/
 
## Install Helm (used to deploy Falco and Gatekeeper)
curl https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3 | bash
 
echo "✅ Tools installed"

YAML

## kind-config.yaml
## Create a cluster with the audit log enabled
## The audit log records every API request — required for security analysis
 
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
- role: control-plane
  kubeadmConfigPatches:
  - |
    kind: ClusterConfiguration
    apiServer:
      extraArgs:
        ## Enable audit logging to file
        audit-log-path: /var/log/kubernetes/audit.log
        ## Log all requests at RequestResponse level
        audit-log-maxage: "30"
        audit-log-maxbackup: "3"
        audit-log-maxsize: "100"
      extraVolumes:
      - name: audit-log
        hostPath: /var/log/kubernetes
        mountPath: /var/log/kubernetes
        readOnly: false
        pathType: DirectoryOrCreate

Bash

## Create the cluster
kind create cluster --config kind-config.yaml --name devsecops-capstone
 
## Verify it is running
kubectl cluster-info --context kind-devsecops-capstone
kubectl get nodes
 
echo "✅ Cluster created"

Layer 1 — RBAC

Understanding Roles, ClusterRoles, and Bindings

RBAC (Role-Based Access Control) is Kubernetes's permission system. It answers one question: "Can this identity perform this action on this resource?"

The four objects:

Role — grants permissions within a single namespace
ClusterRole — grants permissions across the entire cluster
RoleBinding — attaches a Role (or ClusterRole) to a user, group, or ServiceAccount
ClusterRoleBinding — attaches a ClusterRole cluster-wide

The most dangerous RBAC misconfiguration is giving a ServiceAccount more permissions than it needs. A compromised pod inherits the permissions of its ServiceAccount. A pod with cluster-admin permissions is a full cluster compromise.

Auditing existing RBAC permissions

Bash

## Check who has cluster-admin permissions
## This should show only the kubeadm bootstrapper and kube-system accounts
kubectl get clusterrolebindings \
  -o custom-columns='NAME:.metadata.name,ROLE:.roleRef.name,SUBJECTS:.subjects[*].name' \
  | grep cluster-admin
 
## Find all ServiceAccounts that can get secrets cluster-wide
## This is a critical misconfiguration — secrets contain tokens and credentials
kubectl auth can-i get secrets --all-namespaces --as=system:serviceaccount:default:default
 
## List all RBAC bindings in the default namespace
kubectl get rolebindings,clusterrolebindings -n default

Creating the application RBAC structure

The principle you are following is least privilege: every ServiceAccount gets the minimum permissions it needs and nothing more.

YAML

## rbac/namespaces.yaml
## Create separate namespaces for isolation
 
apiVersion: v1
kind: Namespace
metadata:
  name: payment-service
  labels:
    ## This label is used by NetworkPolicy and Pod Security Admission
    app.kubernetes.io/environment: production
    ## Pod Security Admission label — enforced at namespace level
    pod-security.kubernetes.io/enforce: restricted
    pod-security.kubernetes.io/enforce-version: latest
---
apiVersion: v1
kind: Namespace
metadata:
  name: monitoring
  labels:
    app.kubernetes.io/environment: monitoring
    pod-security.kubernetes.io/enforce: baseline
    pod-security.kubernetes.io/enforce-version: latest

YAML

## rbac/payment-service-rbac.yaml
## ServiceAccount with minimal permissions for the payment service
 
apiVersion: v1
kind: ServiceAccount
metadata:
  name: payment-service
  namespace: payment-service
  annotations:
    ## Document why this ServiceAccount exists
    description: "ServiceAccount for the payment processing pods"
## Prevent the default service account token from being auto-mounted
## The payment service does not need to call the Kubernetes API
automountServiceAccountToken: false
---
## The payment service needs to read its own config from a ConfigMap
## That is the ONLY Kubernetes API permission it needs
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: payment-service-reader
  namespace: payment-service
rules:
  ## apiGroups: "" means the core API group (Pods, ConfigMaps, Secrets, etc.)
  - apiGroups: [""]
    resources: ["configmaps"]
    ## Only the specific ConfigMap this service needs
    resourceNames: ["payment-service-config"]
    verbs: ["get", "watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: payment-service-reader-binding
  namespace: payment-service
subjects:
  - kind: ServiceAccount
    name: payment-service
    namespace: payment-service
roleRef:
  kind: Role
  name: payment-service-reader
  apiGroup: rbac.authorization.k8s.io

YAML

## rbac/developer-rbac.yaml
## Developer access: read-only on application namespaces
## Cannot read Secrets (those go through a secrets manager)
 
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: developer-readonly
  namespace: payment-service
rules:
  - apiGroups: ["", "apps", "batch"]
    resources: ["pods", "deployments", "services", "configmaps",
                "replicasets", "jobs", "events"]
    verbs: ["get", "list", "watch"]
  ## Developers can exec into pods for debugging
  - apiGroups: [""]
    resources: ["pods/exec", "pods/log"]
    verbs: ["create", "get"]
  ## Developers explicitly CANNOT:
  ## - read Secrets (not listed here)
  ## - delete resources
  ## - modify RBAC
  ## - access other namespaces
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: developer-readonly-binding
  namespace: payment-service
subjects:
  ## Bind to a group — add engineers to the "developers" group in your IdP
  - kind: Group
    name: developers
    apiGroup: rbac.authorization.k8s.io
roleRef:
  kind: Role
  name: developer-readonly
  apiGroup: rbac.authorization.k8s.io

Bash

## Apply all RBAC resources
kubectl apply -f rbac/
 
## Test the payment-service ServiceAccount permissions
## Should succeed (has permission to get this ConfigMap)
kubectl auth can-i get configmaps/payment-service-config \
  --namespace=payment-service \
  --as=system:serviceaccount:payment-service:payment-service
 
## Should fail (does not have permission)
kubectl auth can-i get secrets \
  --namespace=payment-service \
  --as=system:serviceaccount:payment-service:payment-service
 
## Should fail (cannot access other namespaces)
kubectl auth can-i get pods \
  --namespace=kube-system \
  --as=system:serviceaccount:payment-service:payment-service
 
echo "✅ RBAC layer configured"

Layer 2 — Pod Security Admission

What Pod Security Standards enforce

Pod Security Admission (PSA) replaced the deprecated PodSecurityPolicy in Kubernetes 1.25. It enforces security profiles at the namespace level, controlling what pod configurations are allowed to run.

Three built-in profiles:

Privileged — no restrictions (only for trusted system workloads in kube-system)
Baseline — prevents the most dangerous settings (privileged mode, hostNetwork, hostPID)
Restricted — the most secure profile (requires non-root, drops all capabilities, read-only root filesystem)

Bash

## Test that the restricted policy actually blocks privileged pods
## This should be rejected immediately
 
cat > /tmp/test-privileged-pod.yaml << 'EOF'
apiVersion: v1
kind: Pod
metadata:
  name: test-privileged
  namespace: payment-service
spec:
  containers:
  - name: nginx
    image: nginx:alpine
    securityContext:
      privileged: true
EOF
 
kubectl apply -f /tmp/test-privileged-pod.yaml
## Expected output:
## Error from server (Forbidden): error when creating "/tmp/test-privileged-pod.yaml":
## pods "test-privileged" is forbidden: violates PodSecurity "restricted:latest":
## privileged (container "nginx" must not set securityContext.privileged=true)

YAML

## deployments/payment-service-deployment.yaml
## A compliant pod spec that passes the restricted PSA profile
 
apiVersion: apps/v1
kind: Deployment
metadata:
  name: payment-service
  namespace: payment-service
spec:
  replicas: 2
  selector:
    matchLabels:
      app: payment-service
  template:
    metadata:
      labels:
        app: payment-service
        version: "1.0.0"
    spec:
      serviceAccountName: payment-service
      ## Prevent the service account token from being mounted
      automountServiceAccountToken: false
 
      ## Pod-level security context
      securityContext:
        ## Run as a non-root user
        runAsNonRoot: true
        runAsUser: 10001
        runAsGroup: 10001
        ## Prevent privilege escalation via setuid binaries
        seccompProfile:
          type: RuntimeDefault
 
      containers:
      - name: payment-service
        image: ghcr.io/razorpay/payment-service:abc1234
        ports:
        - containerPort: 8080
 
        ## Container-level security context — restricted profile requires all of these
        securityContext:
          ## Container cannot escalate privileges (sudo, setuid)
          allowPrivilegeEscalation: false
          readOnlyRootFilesystem: true
          ## Drop ALL Linux capabilities and add back only what is needed
          ## Most web services need zero capabilities
          capabilities:
            drop: ["ALL"]
 
        resources:
          requests:
            cpu: "100m"
            memory: "128Mi"
          limits:
            cpu: "500m"
            memory: "256Mi"
 
        ## readOnlyRootFilesystem=true means the app cannot write anywhere by default
        ## Mount writable volumes only for directories that need writes
        volumeMounts:
        - name: tmp-dir
          mountPath: /tmp
        - name: exports-dir
          mountPath: /exports
 
      volumes:
      ## emptyDir creates a temporary in-memory filesystem
      - name: tmp-dir
        emptyDir: {}
      - name: exports-dir
        emptyDir: {}

Bash

## Apply the compliant deployment
kubectl apply -f deployments/payment-service-deployment.yaml
 
## Verify it started successfully
kubectl get pods -n payment-service
## Expected: 2 pods in Running state
 
echo "✅ Pod Security Admission layer configured"

Layer 3 — Network Policies

The default network problem

By default, every pod in a Kubernetes cluster can reach every other pod on any port. A compromised frontend pod can connect directly to your database pod on port 5432. An attacker who escapes from the payment-service can probe every service in the cluster.

NetworkPolicy is a Kubernetes firewall that controls pod-to-pod traffic. It works at the IP and port level. Without a NetworkPolicy, all traffic is allowed. With a NetworkPolicy, all traffic not explicitly allowed is denied.

Remember
NetworkPolicy requires a CNI (Container Network Interface) plugin that supports it. Flannel does not. Calico, Cilium, and Weave all do. Kind uses Kindnet by default — install Calico for NetworkPolicy support in this capstone.

Bash

## Install Calico CNI for NetworkPolicy support in kind
kubectl apply -f https://raw.githubusercontent.com/projectcalico/calico/v3.27.0/manifests/calico.yaml
 
## Wait for Calico to be ready
kubectl rollout status daemonset/calico-node -n kube-system --timeout=120s
 
echo "✅ Calico CNI installed — NetworkPolicies now enforced"

YAML

## network-policies/default-deny-all.yaml
## The most important policy: deny everything unless explicitly allowed.
## Apply this first, then add allow policies for legitimate traffic.
 
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny-all
  namespace: payment-service
spec:
  ## podSelector: {} means this policy applies to ALL pods in the namespace
  podSelector: {}
  ## policyTypes with both Ingress and Egress means deny all traffic in both directions
  policyTypes:
  - Ingress
  - Egress

YAML

## network-policies/allow-payment-ingress.yaml
## Allow external traffic to reach the payment service from the ingress controller
 
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-payment-ingress
  namespace: payment-service
spec:
  podSelector:
    matchLabels:
      app: payment-service
  policyTypes:
  - Ingress
  ingress:
  - from:
    ## Only allow traffic from the ingress-nginx namespace
    - namespaceSelector:
        matchLabels:
          kubernetes.io/metadata.name: ingress-nginx
      ## Within that namespace, only from pods labelled as the ingress controller
      podSelector:
        matchLabels:
          app.kubernetes.io/name: ingress-nginx
    ports:
    - protocol: TCP
      port: 8080

YAML

## network-policies/allow-payment-egress.yaml
## Allow the payment service to make outbound connections it needs
 
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-payment-egress
  namespace: payment-service
spec:
  podSelector:
    matchLabels:
      app: payment-service
  policyTypes:
  - Egress
  egress:
  ## Allow DNS resolution — without this the service cannot resolve any hostnames
  - ports:
    - protocol: UDP
      port: 53
    - protocol: TCP
      port: 53
 
  ## Allow connecting to the database namespace only
  - to:
    - namespaceSelector:
        matchLabels:
          kubernetes.io/metadata.name: database
      podSelector:
        matchLabels:
          app: postgresql
    ports:
    - protocol: TCP
      port: 5432
 
  ## Allow connecting to external payment processors (Razorpay, NPCI)
  ## In production: use CIDR ranges for specific IPs instead of any
  - ports:
    - protocol: TCP
      port: 443

YAML

## network-policies/allow-monitoring-scrape.yaml
## Allow Prometheus to scrape metrics from the payment service
 
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-monitoring-scrape
  namespace: payment-service
spec:
  podSelector:
    matchLabels:
      app: payment-service
  policyTypes:
  - Ingress
  ingress:
  - from:
    - namespaceSelector:
        matchLabels:
          kubernetes.io/metadata.name: monitoring
      podSelector:
        matchLabels:
          app.kubernetes.io/name: prometheus
    ports:
    - protocol: TCP
      port: 9090

Bash

## Apply all network policies
kubectl apply -f network-policies/
 
## Test that the policies work
 
## Deploy a test pod in the payment-service namespace
kubectl run test-pod \
  --image=busybox:1.35 \
  --restart=Never \
  --namespace=payment-service \
  -- sleep 3600
 
## Should FAIL — default deny blocks inter-pod traffic
kubectl exec -n payment-service test-pod -- \
  wget -T 2 -q http://kubernetes.default.svc.cluster.local
## Expected: wget: download timed out
 
## Should SUCCEED — DNS is explicitly allowed
kubectl exec -n payment-service test-pod -- \
  nslookup kubernetes.default.svc.cluster.local
## Expected: successful DNS resolution
 
echo "✅ Network Policy layer configured"

Layer 4 — OPA Gatekeeper

Writing custom admission policies

OPA Gatekeeper is an admission controller that validates Kubernetes resource requests against custom policies before they are created. It uses Rego, a policy language, to define rules.

While Pod Security Admission handles common security profiles, Gatekeeper handles your organization-specific rules:

All production containers must have resource limits set
Container images must come from the approved registry only
All deployments must have at least 2 replicas
Labels for team and cost center are required on every namespace

Bash

## Install OPA Gatekeeper
kubectl apply -f https://raw.githubusercontent.com/open-policy-agent/gatekeeper/v3.14.0/deploy/gatekeeper.yaml
 
## Wait for Gatekeeper to be ready
kubectl rollout status deployment/gatekeeper-controller-manager \
  -n gatekeeper-system --timeout=120s
 
echo "✅ OPA Gatekeeper installed"

YAML

## gatekeeper/allowed-registries-template.yaml
## Define the policy template — this is the Rego logic
## A ConstraintTemplate defines what the policy checks.
## A Constraint (next file) configures where and how it applies.
 
apiVersion: templates.gatekeeper.sh/v1
kind: ConstraintTemplate
metadata:
  name: k8sallowedregistries
  annotations:
    description: "Require container images to come from approved registries only"
spec:
  crd:
    spec:
      names:
        kind: K8sAllowedRegistries
      ## These are the parameters you configure in the Constraint
      validation:
        openAPIV3Schema:
          type: object
          properties:
            registries:
              type: array
              items:
                type: string
 
  targets:
    - target: admission.k8s.gatekeeper.sh
      rego: |
        package k8sallowedregistries
        violation[{"msg": msg}] {
          ## Get every container in the pod spec (including init containers)
          container := input.review.object.spec.containers[_]
          ## Check if the image starts with any of the allowed registries
          not starts_with_allowed(container.image)
          msg := sprintf(
            "Container '%v' uses image '%v' which is not from an approved registry. Allowed: %v",
            [container.name, container.image, input.parameters.registries]
          )
        }
 
        ## Check initContainers too
        violation[{"msg": msg}] {
          container := input.review.object.spec.initContainers[_]
          not starts_with_allowed(container.image)
          msg := sprintf(
            "Init container '%v' uses image '%v' not from approved registry",
            [container.name, container.image]
          )
        }
 
        starts_with_allowed(image) {
          ## startswith checks if image begins with one of the approved registry prefixes
          startswith(image, input.parameters.registries[_])
        }

YAML

## gatekeeper/allowed-registries-constraint.yaml
## Apply the policy — specify allowed registries and which namespaces to check
 
apiVersion: constraints.gatekeeper.sh/v1beta1
kind: K8sAllowedRegistries
metadata:
  name: allowed-registries
spec:
  ## enforcementAction: deny = block the resource if policy fails
  ## Use "warn" first when rolling out to see what would be blocked
  enforcementAction: deny
  match:
    kinds:
      - apiGroups: [""]
        kinds: ["Pod"]
    ## Apply to all namespaces except kube-system and gatekeeper-system
    excludedNamespaces: ["kube-system", "gatekeeper-system"]
  parameters:
    registries:
      - "ghcr.io/razorpay/"
      - "registry.k8s.io/"
      - "docker.io/library/"

YAML

## gatekeeper/require-resource-limits-template.yaml
## Policy: all containers must have CPU and memory limits
 
apiVersion: templates.gatekeeper.sh/v1
kind: ConstraintTemplate
metadata:
  name: k8srequirelimits
  annotations:
    description: "Require all containers to have CPU and memory resource limits"
spec:
  crd:
    spec:
      names:
        kind: K8sRequireLimits
  targets:
    - target: admission.k8s.gatekeeper.sh
      rego: |
        package k8srequirelimits
        violation[{"msg": msg}] {
          container := input.review.object.spec.containers[_]
          ## Check if memory limit is missing
          not container.resources.limits.memory
          msg := sprintf(
            "Container '%v' is missing memory limit. Add resources.limits.memory.",
            [container.name]
          )
        }
 
        violation[{"msg": msg}] {
          container := input.review.object.spec.containers[_]
          ## Check if CPU limit is missing
          not container.resources.limits.cpu
          msg := sprintf(
            "Container '%v' is missing CPU limit. Add resources.limits.cpu.",
            [container.name]
          )
        }

YAML

## gatekeeper/require-resource-limits-constraint.yaml
apiVersion: constraints.gatekeeper.sh/v1beta1
kind: K8sRequireLimits
metadata:
  name: require-resource-limits
spec:
  enforcementAction: deny
  match:
    kinds:
      - apiGroups: [""]
        kinds: ["Pod"]
    excludedNamespaces: ["kube-system", "gatekeeper-system"]

Bash

## Apply all Gatekeeper policies
kubectl apply -f gatekeeper/
 
## Test the registry policy — should be rejected
cat > /tmp/test-registry.yaml << 'EOF'
apiVersion: v1
kind: Pod
metadata:
  name: test-registry
  namespace: payment-service
spec:
  containers:
  - name: app
    image: nginx:latest
    resources:
      limits:
        cpu: "100m"
        memory: "128Mi"
EOF
 
kubectl apply -f /tmp/test-registry.yaml
## Expected:
## Error from server (Forbidden): admission webhook "validation.gatekeeper.sh" denied:
## Container 'app' uses image 'nginx:latest' which is not from an approved registry.
## Allowed: ["ghcr.io/razorpay/", "registry.k8s.io/", "docker.io/library/"]
 
## Test the resource limits policy — should be rejected
cat > /tmp/test-no-limits.yaml << 'EOF'
apiVersion: v1
kind: Pod
metadata:
  name: test-no-limits
  namespace: payment-service
spec:
  containers:
  - name: app
    image: ghcr.io/razorpay/payment-service:latest
EOF
 
kubectl apply -f /tmp/test-no-limits.yaml
## Expected error about missing resource limits
 
echo "✅ OPA Gatekeeper policies active"

Layer 5 — Falco Runtime Detection

Detecting threats that slipped past admission

The four previous layers are all preventive controls. They stop bad configurations before they run. Falco is a detective control — it watches what is actually happening inside running containers and alerts when something suspicious occurs.

Falco uses Linux kernel syscall tracing (via eBPF) to see every system call made by every container. Its rule engine checks those syscalls against patterns that indicate attacks:

A shell was spawned inside a container (common first step after container escape)
A binary was written to a directory that should be read-only
A container opened a sensitive file like /etc/shadow or /proc/*/mem
A process made a network connection to an unexpected port

Bash

## Add the Falco Helm repository
helm repo add falcosecurity https://falcosecurity.github.io/charts
helm repo update
 
## Install Falco with eBPF driver (requires kernel headers)
## driver.kind: ebpf = uses eBPF instead of a kernel module (safer, more portable)
helm install falco falcosecurity/falco \
  --namespace falco \
  --create-namespace \
  --set driver.kind=ebpf \
  --set falcosidekick.enabled=true \
  --set falcosidekick.webui.enabled=true
 
## Wait for Falco to be ready
kubectl rollout status daemonset/falco -n falco --timeout=180s
 
echo "✅ Falco installed"

YAML

## falco/custom-rules.yaml
## Custom rules specific to the payment service
## Add these to the Falco ConfigMap to detect payment-specific threats
 
customRules:
  payment-service-rules.yaml: |-
    ## Rule 1: Detect shell execution inside payment-service containers
    ## A legitimate payment service never needs a shell at runtime
    - rule: Shell Spawned in Payment Service Container
      desc: >
        A shell was executed inside a payment-service container.
        This is never expected in production — indicates container escape or supply chain attack.
      condition: >
        spawned_process
        and container.name contains "payment-service"
        and proc.name in (shell_binaries)
      output: >
        Shell spawned in payment container
        (user=%user.name container=%container.name
         image=%container.image.repository:%container.image.tag
         shell=%proc.name parent=%proc.pname cmdline=%proc.cmdline)
      priority: CRITICAL
      tags: [payment, container, shell]
    ## Rule 2: Detect unexpected outbound connections from payment service
    ## The payment service should only connect to the database and Razorpay API
    - rule: Unexpected Network Connection from Payment Service
      desc: >
        The payment-service pod made a network connection to an unexpected destination.
        Could indicate data exfiltration or command-and-control communication.
      condition: >
        outbound
        and container.name contains "payment-service"
        and not fd.sip in (payment_allowed_ips)
        and not fd.sport in (payment_allowed_ports)
      output: >
        Unexpected outbound connection from payment service
        (user=%user.name container=%container.name
         destination_ip=%fd.sip destination_port=%fd.sport
         process=%proc.name cmdline=%proc.cmdline)
      priority: HIGH
      tags: [payment, network, exfiltration]
 
    ## Rule 3: Detect cryptocurrency mining processes
    ## Mining processes have distinct CPU usage patterns and connect to mining pools
    - rule: Crypto Miner Process Detected
      desc: >
        A process associated with cryptocurrency mining was started.
        Indicates unauthorized resource use — a compromised container running a miner.
      condition: >
        spawned_process
        and proc.name in (crypto_miners)
      output: >
        Crypto miner process started
        (user=%user.name host=%evt.hostname
         container=%container.name image=%container.image.repository
         process=%proc.name cmdline=%proc.cmdline)
      priority: CRITICAL
      tags: [cryptomining, resource-abuse]
 
    ## Macros used in the rules above
    - macro: shell_binaries
      condition: >
        proc.name in (bash, sh, zsh, fish, dash, ksh, tcsh, csh)
    - list: crypto_miners
      items: [xmrig, minerd, cpuminer, ethminer, t-rex, gminer, nbminer]
 
    - list: payment_allowed_ips
      items: ["10.0.0.0/8", "172.16.0.0/12", "192.168.0.0/16"]
 
    - list: payment_allowed_ports
      items: [5432, 443, 53]

Bash

## Apply the custom rules via ConfigMap
kubectl apply -f falco/custom-rules.yaml
 
## Restart Falco to pick up the new rules
kubectl rollout restart daemonset/falco -n falco
 
## Test that Falco detects shell execution
## Run a shell inside a payment-service pod
kubectl exec -n payment-service \
  $(kubectl get pods -n payment-service -l app=payment-service -o name | head -1) \
  -- /bin/sh -c "echo 'test'"
 
## Check Falco logs for the alert
kubectl logs -n falco daemonset/falco | grep "Shell Spawned" | tail -5
## Expected output:
## 10:30:45.123456789: Critical Shell spawned in payment container
## (user=root container=payment-service image=ghcr.io/razorpay/payment-service:abc1234
##  shell=sh parent=kubectl cmdline=sh -c echo 'test')
 
echo "✅ Falco runtime detection working"

Layer 6 — CIS Compliance Scanning with kube-bench

Automated compliance auditing

kube-bench is an open-source tool that checks Kubernetes cluster configuration against the CIS Kubernetes Benchmark — a set of security recommendations from the Center for Internet Security. It checks everything from API server flags to kubelet configuration to etcd encryption.

Running kube-bench gives you a prioritized list of what to fix to meet compliance standards like SOC 2, PCI-DSS, and ISO 27001.

Bash

## Run kube-bench as a Kubernetes Job
## The Job runs kube-bench inside the cluster with appropriate permissions
 
cat > /tmp/kube-bench-job.yaml << 'EOF'
apiVersion: batch/v1
kind: Job
metadata:
  name: kube-bench
  namespace: default
spec:
  template:
    spec:
      hostPID: true
      serviceAccountName: default
      containers:
      - name: kube-bench
        image: aquasec/kube-bench:v0.7.2
        command: ["kube-bench", "--json", "--outputfile", "/tmp/results.json"]
        volumeMounts:
        - name: var-lib-etcd
          mountPath: /var/lib/etcd
          readOnly: true
        - name: var-lib-kubelet
          mountPath: /var/lib/kubelet
          readOnly: true
        - name: etc-kubernetes
          mountPath: /etc/kubernetes
          readOnly: true
      volumes:
      - name: var-lib-etcd
        hostPath:
          path: /var/lib/etcd
      - name: var-lib-kubelet
        hostPath:
          path: /var/lib/kubelet
      - name: etc-kubernetes
        hostPath:
          path: /etc/kubernetes
      restartPolicy: Never
EOF
 
kubectl apply -f /tmp/kube-bench-job.yaml
 
## Wait for completion
kubectl wait --for=condition=complete job/kube-bench --timeout=120s
 
## View the results — look at FAIL entries first
kubectl logs job/kube-bench | grep -E "^\[FAIL\]" | head -20

Bash

## Parse the JSON output for a scorecard
kubectl logs job/kube-bench | python3 -c "
import json, sys
results = json.load(sys.stdin)
total = pass_count = fail_count = warn_count = 0
for control in results.get('Controls', []):
    for test in control.get('tests', []):
        for result in test.get('results', []):
            total += 1
            status = result.get('status', '')
            if status == 'PASS': pass_count += 1
            elif status == 'FAIL': fail_count += 1
            elif status == 'WARN': warn_count += 1
 
print(f'CIS Benchmark Scorecard:')
print(f'  Total checks:  {total}')
print(f'  Passed:        {pass_count} ({pass_count/total*100:.1f}%)')
print(f'  Failed:        {fail_count} ({fail_count/total*100:.1f}%)')
print(f'  Warnings:      {warn_count} ({warn_count/total*100:.1f}%)')
print()
print('Focus on FAIL items first. Aim for < 5 FAIL in production.')
"

Validating All Six Layers Together

Bash

## Full end-to-end validation script
 
echo "=== DevSecOps Kubernetes Hardening — Validation ==="
echo ""
 
## Layer 1: RBAC
echo "--- Layer 1: RBAC ---"
## ServiceAccount cannot get secrets
RESULT=$(kubectl auth can-i get secrets \
  --namespace=payment-service \
  --as=system:serviceaccount:payment-service:payment-service 2>&1)
if echo "$RESULT" | grep -q "no"; then
  echo "✅ RBAC: payment-service cannot get secrets"
else
  echo "❌ RBAC: FAIL — payment-service can get secrets (unexpected)"
fi
 
## Layer 2: Pod Security Admission
echo ""
echo "--- Layer 2: Pod Security Admission ---"
RESULT=$(kubectl apply -f /tmp/test-privileged-pod.yaml 2>&1)
if echo "$RESULT" | grep -q "Forbidden"; then
  echo "✅ PSA: privileged pods are blocked in payment-service namespace"
else
  echo "❌ PSA: FAIL — privileged pod was not blocked"
fi
 
## Layer 3: NetworkPolicy
echo ""
echo "--- Layer 3: NetworkPolicy ---"
RESULT=$(kubectl exec -n payment-service test-pod -- \
  wget -T 2 -q http://kube-dns.kube-system.svc.cluster.local 2>&1)
if echo "$RESULT" | grep -q "timed out\|Connection refused"; then
  echo "✅ NetworkPolicy: cross-namespace traffic is blocked"
else
  echo "❌ NetworkPolicy: FAIL — unexpected connection succeeded"
fi
 
## Layer 4: OPA Gatekeeper
echo ""
echo "--- Layer 4: OPA Gatekeeper ---"
RESULT=$(kubectl apply -f /tmp/test-registry.yaml 2>&1)
if echo "$RESULT" | grep -q "denied"; then
  echo "✅ Gatekeeper: unapproved registry is blocked"
else
  echo "❌ Gatekeeper: FAIL — pod from unapproved registry was allowed"
fi
 
## Layer 5: Falco
echo ""
echo "--- Layer 5: Falco ---"
FALCO_RUNNING=$(kubectl get pods -n falco -l app.kubernetes.io/name=falco \
  --field-selector=status.phase=Running --no-headers | wc -l)
if [ "$FALCO_RUNNING" -gt 0 ]; then
  echo "✅ Falco: $FALCO_RUNNING Falco pod(s) running and monitoring"
else
  echo "❌ Falco: FAIL — no Falco pods running"
fi
 
## Layer 6: kube-bench
echo ""
echo "--- Layer 6: CIS Compliance ---"
FAIL_COUNT=$(kubectl logs job/kube-bench 2>/dev/null | grep -c "^\[FAIL\]" || echo "N/A")
echo "ℹ️  kube-bench FAIL count: $FAIL_COUNT (target: <10 for new clusters)"
 
echo ""
echo "=== Validation Complete ==="

Production Checklist

Bash

## ─── RBAC ────────────────────────────────────────────────────
## No service account with cluster-admin except system accounts
kubectl get clusterrolebindings -o json | \
  python3 -c "
import json, sys
data = json.load(sys.stdin)
for b in data['items']:
    if b['roleRef']['name'] == 'cluster-admin':
        subjects = b.get('subjects', [])
        for s in subjects:
            if not s.get('namespace', '').startswith('kube-'):
                print(f'⚠️  Non-system cluster-admin: {s}')
print('✅ RBAC cluster-admin check complete')
"
 
## ─── Pod Security ────────────────────────────────────────────
## Verify restricted PSA is applied to production namespaces
kubectl get namespaces -o json | \
  python3 -c "
import json, sys
data = json.load(sys.stdin)
for ns in data['items']:
    labels = ns['metadata'].get('labels', {})
    name = ns['metadata']['name']
    if name in ['payment-service', 'default']:
        level = labels.get('pod-security.kubernetes.io/enforce', 'NONE')
        status = '✅' if level == 'restricted' else '⚠️'
        print(f'{status} {name}: PSA={level}')
"
 
## ─── NetworkPolicy ────────────────────────────────────────────
## Every non-system namespace should have a default-deny policy
for ns in payment-service; do
  COUNT=$(kubectl get networkpolicies -n $ns \
    -o jsonpath='{.items[*].metadata.name}' | tr ' ' '\n' | \
    grep -c default-deny || echo 0)
  if [ "$COUNT" -gt 0 ]; then
    echo "✅ NetworkPolicy: default-deny present in $ns"
  else
    echo "❌ NetworkPolicy: NO default-deny in $ns"
  fi
done
 
## ─── Gatekeeper ───────────────────────────────────────────────
kubectl get constraints 2>/dev/null | grep -v "^NAME" | while read line; do
  echo "✅ Gatekeeper constraint active: $line"
done
 
## ─── Falco ────────────────────────────────────────────────────
kubectl get pods -n falco --field-selector=status.phase=Running --no-headers | \
  awk '{print "✅ Falco running: " $1}'
 
echo "✅ All production checks complete"

Common Production Mistakes

Giving every new ServiceAccount the default ServiceAccount permissions. Kubernetes automatically creates a default ServiceAccount in every namespace, and newly created pods use it unless specified otherwise. If developers do not explicitly set serviceAccountName in their pod spec, their pod gets whatever permissions the default ServiceAccount has. Set automountServiceAccountToken: false on the default ServiceAccount in every namespace and require explicit ServiceAccount assignments.

Setting NetworkPolicy enforcementAction: warn and never switching to deny. The warn mode is appropriate when first rolling out policies to a cluster with existing workloads — you need to see what would be blocked before actually blocking it. The mistake is leaving it in warn mode permanently. Warnings that do not block are documentation, not security. Set a calendar reminder for two weeks after rollout to review warnings and switch to deny.

Writing Gatekeeper rules that are too broad and breaking legitimate system workloads. A policy that applies to all namespaces including kube-system will break Kubernetes itself — system pods use images from registry.k8s.io which may not match your allowed registry list. Always add excludedNamespaces: ["kube-system", "gatekeeper-system", "cert-manager"] to every Constraint. Test every new policy in enforcementAction: warn first.

Not alerting on Falco CRITICAL events. Falco generates events continuously. Without routing CRITICAL events to Slack or PagerDuty, they are logs that nobody reads. Install Falcosidekick with a Slack webhook configured on day one. A CRITICAL Falco event — shell in container, sensitive file read, crypto miner — should page the on-call engineer within 2 minutes, not be discovered during a weekly log review.

Treating kube-bench WARN as acceptable. kube-bench produces three statuses: PASS, FAIL, and WARN. WARN means the check requires manual investigation because the correct setting depends on your environment. Teams often focus only on FAIL items and ignore all WARNs. Many WARNs represent real security gaps — audit log configuration, anonymous API access, insecure kubelet settings. Review every WARN and document your decision on each one.

Running Falco with the kernel module driver instead of eBPF in production. The kernel module approach requires loading a custom kernel module, which increases the attack surface and can cause system instability if the module crashes. eBPF is sandboxed by the Linux kernel's verifier, cannot crash the system, and requires no kernel module installation. Always use driver.kind: ebpf in production clusters.

Quick Reference

Layer	Tool	What It Controls	Where Configured
RBAC	kubectl	API access permissions	ClusterRole / Role YAML
Pod Security	PSA	Pod spec restrictions	Namespace labels
Network	Calico	Pod-to-pod traffic	NetworkPolicy YAML
Admission	Gatekeeper	Custom policy gates	ConstraintTemplate + Constraint
Runtime	Falco	Live container behavior	Falco rules ConfigMap
Compliance	kube-bench	CIS Benchmark score	Scheduled Job

Command	What It Does
`kubectl auth can-i ACTION --as=system:serviceaccount:NS:SA`	Test RBAC permissions
`kubectl get networkpolicies -n NAMESPACE`	List active network policies
`kubectl get constraints`	List active Gatekeeper policies
`kubectl logs -n falco daemonset/falco \| grep CRITICAL`	Check Falco alerts
`kubectl logs job/kube-bench \| grep FAIL`	View CIS benchmark failures
`kubectl label ns NAMESPACE pod-security.kubernetes.io/enforce=restricted`	Apply PSA to namespace

Videos & Guides

No external walkthroughs, video tutorials, or reference files attached to this blueprint yet.