A fintech startup running on Kubernetes had been in production for eight months. Their application was secure. Their code had no vulnerabilities. Their team was competent. Then a developer accidentally pushed a pod manifest that had `hostPID: true` set — a leftover from debugging. The pod started successfully. An attacker who found a remote code execution vulnerability in the application could now escape the container and see every process running on the Kubernetes node — including processes from other customers' workloads on the same node. The vulnerability was not in the code. It was in a missing policy. Nobody had told Kubernetes "never allow `hostPID: true` in this cluster." Kubernetes does exactly what you tell it to do. If you do not tell it what to refuse, it refuses nothing. This capstone hardens a Kubernetes cluster using six layers of security controls. When you finish, a developer cannot accidentally — or deliberately — deploy a privileged container, access another service they should not, escalate their own permissions, or run a cryptocurrency miner without an alert firing within 60 seconds. The six layers: ┌─────────────────────────────────────────────────────────────┐ │ Layer 1: RBAC — Who can do what in the cluster │ │ Layer 2: Pod Security Admission — What pods can run │ │ Layer 3: NetworkPolicies — What can talk to what │ │ Layer 4: OPA Gatekeeper — Custom admission policies │ │ Layer 5: Falco — Runtime threat detection │ │ Layer 6: kube-bench — Automated CIS compliance scanning │ └─────────────────────────────────────────────────────────────┘
### What Kubernetes exposes by default Kubernetes is secure by design in some areas and dangerously permissive in others. Understanding what is locked by default and what you need to lock yourself is the foundation of cluster hardening. **Locked by default:** * etcd encryption at rest (if configured during cluster bootstrap) * API server TLS * Service account token mounting (can be disabled) **Open by default — you must close these:** * All pods in a namespace can reach all other pods in the cluster on any port * A pod can request root (UID 0) inside the container * A pod can mount the host filesystem * Any authenticated user who can create pods can potentially escalate to cluster-admin * No runtime monitoring — if a container starts a shell or writes a binary, nothing alerts ```bash ## Set up a local cluster for this capstone ## Use kind (Kubernetes IN Docker) — it runs a full K8s cluster locally ## Install kind curl -Lo ./kind https://kind.sigs.k8s.io/dl/v0.22.0/kind-linux-amd64 chmod +x ./kind && sudo mv ./kind /usr/local/bin/kind ## Install kubectl curl -LO "https://dl.k8s.io/release/$(curl -L -s \ https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl" chmod +x kubectl && sudo mv kubectl /usr/local/bin/ ## Install Helm (used to deploy Falco and Gatekeeper) curl https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3 | bash echo "✅ Tools installed" ``` ```yaml ## kind-config.yaml ## Create a cluster with the audit log enabled ## The audit log records every API request — required for security analysis kind: Cluster apiVersion: kind.x-k8s.io/v1alpha4 nodes: - role: control-plane kubeadmConfigPatches: - | kind: ClusterConfiguration apiServer: extraArgs: ## Enable audit logging to file audit-log-path: /var/log/kubernetes/audit.log ## Log all requests at RequestResponse level audit-log-maxage: "30" audit-log-maxbackup: "3" audit-log-maxsize: "100" extraVolumes: - name: audit-log hostPath: /var/log/kubernetes mountPath: /var/log/kubernetes readOnly: false pathType: DirectoryOrCreate ``` ```bash ## Create the cluster kind create cluster --config kind-config.yaml --name devsecops-capstone ## Verify it is running kubectl cluster-info --context kind-devsecops-capstone kubectl get nodes echo "✅ Cluster created" ```
### Understanding Roles, ClusterRoles, and Bindings **RBAC (Role-Based Access Control)** is Kubernetes's permission system. It answers one question: "Can this identity perform this action on this resource?" The four objects: * **Role** — grants permissions within a single namespace * **ClusterRole** — grants permissions across the entire cluster * **RoleBinding** — attaches a Role (or ClusterRole) to a user, group, or ServiceAccount * **ClusterRoleBinding** — attaches a ClusterRole cluster-wide The most dangerous RBAC misconfiguration is giving a ServiceAccount more permissions than it needs. A compromised pod inherits the permissions of its ServiceAccount. A pod with `cluster-admin` permissions is a full cluster compromise. ### Auditing existing RBAC permissions ```bash ## Check who has cluster-admin permissions ## This should show only the kubeadm bootstrapper and kube-system accounts kubectl get clusterrolebindings \ -o custom-columns='NAME:.metadata.name,ROLE:.roleRef.name,SUBJECTS:.subjects[*].name' \ | grep cluster-admin ## Find all ServiceAccounts that can get secrets cluster-wide ## This is a critical misconfiguration — secrets contain tokens and credentials kubectl auth can-i get secrets --all-namespaces --as=system:serviceaccount:default:default ## List all RBAC bindings in the default namespace kubectl get rolebindings,clusterrolebindings -n default ``` ### Creating the application RBAC structure The principle you are following is **least privilege**: every ServiceAccount gets the minimum permissions it needs and nothing more. ```yaml ## rbac/namespaces.yaml ## Create separate namespaces for isolation apiVersion: v1 kind: Namespace metadata: name: payment-service labels: ## This label is used by NetworkPolicy and Pod Security Admission app.kubernetes.io/environment: production ## Pod Security Admission label — enforced at namespace level pod-security.kubernetes.io/enforce: restricted pod-security.kubernetes.io/enforce-version: latest --- apiVersion: v1 kind: Namespace metadata: name: monitoring labels: app.kubernetes.io/environment: monitoring pod-security.kubernetes.io/enforce: baseline pod-security.kubernetes.io/enforce-version: latest ``` ```yaml ## rbac/payment-service-rbac.yaml ## ServiceAccount with minimal permissions for the payment service apiVersion: v1 kind: ServiceAccount metadata: name: payment-service namespace: payment-service annotations: ## Document why this ServiceAccount exists description: "ServiceAccount for the payment processing pods" ## Prevent the default service account token from being auto-mounted ## The payment service does not need to call the Kubernetes API automountServiceAccountToken: false --- ## The payment service needs to read its own config from a ConfigMap ## That is the ONLY Kubernetes API permission it needs apiVersion: rbac.authorization.k8s.io/v1 kind: Role metadata: name: payment-service-reader namespace: payment-service rules: ## apiGroups: "" means the core API group (Pods, ConfigMaps, Secrets, etc.) - apiGroups: [""] resources: ["configmaps"] ## Only the specific ConfigMap this service needs resourceNames: ["payment-service-config"] verbs: ["get", "watch"] --- apiVersion: rbac.authorization.k8s.io/v1 kind: RoleBinding metadata: name: payment-service-reader-binding namespace: payment-service subjects: - kind: ServiceAccount name: payment-service namespace: payment-service roleRef: kind: Role name: payment-service-reader apiGroup: rbac.authorization.k8s.io ``` ```yaml ## rbac/developer-rbac.yaml ## Developer access: read-only on application namespaces ## Cannot read Secrets (those go through a secrets manager) apiVersion: rbac.authorization.k8s.io/v1 kind: Role metadata: name: developer-readonly namespace: payment-service rules: - apiGroups: ["", "apps", "batch"] resources: ["pods", "deployments", "services", "configmaps", "replicasets", "jobs", "events"] verbs: ["get", "list", "watch"] ## Developers can exec into pods for debugging - apiGroups: [""] resources: ["pods/exec", "pods/log"] verbs: ["create", "get"] ## Developers explicitly CANNOT: ## - read Secrets (not listed here) ## - delete resources ## - modify RBAC ## - access other namespaces --- apiVersion: rbac.authorization.k8s.io/v1 kind: RoleBinding metadata: name: developer-readonly-binding namespace: payment-service subjects: ## Bind to a group — add engineers to the "developers" group in your IdP - kind: Group name: developers apiGroup: rbac.authorization.k8s.io roleRef: kind: Role name: developer-readonly apiGroup: rbac.authorization.k8s.io ``` ```bash ## Apply all RBAC resources kubectl apply -f rbac/ ## Test the payment-service ServiceAccount permissions ## Should succeed (has permission to get this ConfigMap) kubectl auth can-i get configmaps/payment-service-config \ --namespace=payment-service \ --as=system:serviceaccount:payment-service:payment-service ## Should fail (does not have permission) kubectl auth can-i get secrets \ --namespace=payment-service \ --as=system:serviceaccount:payment-service:payment-service ## Should fail (cannot access other namespaces) kubectl auth can-i get pods \ --namespace=kube-system \ --as=system:serviceaccount:payment-service:payment-service echo "✅ RBAC layer configured" ```
### What Pod Security Standards enforce **Pod Security Admission (PSA)** replaced the deprecated PodSecurityPolicy in Kubernetes 1.25. It enforces security profiles at the namespace level, controlling what pod configurations are allowed to run. Three built-in profiles: * **Privileged** — no restrictions (only for trusted system workloads in kube-system) * **Baseline** — prevents the most dangerous settings (privileged mode, hostNetwork, hostPID) * **Restricted** — the most secure profile (requires non-root, drops all capabilities, read-only root filesystem) ```bash ## Test that the restricted policy actually blocks privileged pods ## This should be rejected immediately cat > /tmp/test-privileged-pod.yaml << 'EOF' apiVersion: v1 kind: Pod metadata: name: test-privileged namespace: payment-service spec: containers: - name: nginx image: nginx:alpine securityContext: privileged: true EOF kubectl apply -f /tmp/test-privileged-pod.yaml ## Expected output: ## Error from server (Forbidden): error when creating "/tmp/test-privileged-pod.yaml": ## pods "test-privileged" is forbidden: violates PodSecurity "restricted:latest": ## privileged (container "nginx" must not set securityContext.privileged=true) ``` ```yaml ## deployments/payment-service-deployment.yaml ## A compliant pod spec that passes the restricted PSA profile apiVersion: apps/v1 kind: Deployment metadata: name: payment-service namespace: payment-service spec: replicas: 2 selector: matchLabels: app: payment-service template: metadata: labels: app: payment-service version: "1.0.0" spec: serviceAccountName: payment-service ## Prevent the service account token from being mounted automountServiceAccountToken: false ## Pod-level security context securityContext: ## Run as a non-root user runAsNonRoot: true runAsUser: 10001 runAsGroup: 10001 ## Prevent privilege escalation via setuid binaries seccompProfile: type: RuntimeDefault containers: - name: payment-service image: ghcr.io/razorpay/payment-service:abc1234 ports: - containerPort: 8080 ## Container-level security context — restricted profile requires all of these securityContext: ## Container cannot escalate privileges (sudo, setuid) allowPrivilegeEscalation: false readOnlyRootFilesystem: true ## Drop ALL Linux capabilities and add back only what is needed ## Most web services need zero capabilities capabilities: drop: ["ALL"] resources: requests: cpu: "100m" memory: "128Mi" limits: cpu: "500m" memory: "256Mi" ## readOnlyRootFilesystem=true means the app cannot write anywhere by default ## Mount writable volumes only for directories that need writes volumeMounts: - name: tmp-dir mountPath: /tmp - name: exports-dir mountPath: /exports volumes: ## emptyDir creates a temporary in-memory filesystem - name: tmp-dir emptyDir: {} - name: exports-dir emptyDir: {} ``` ```bash ## Apply the compliant deployment kubectl apply -f deployments/payment-service-deployment.yaml ## Verify it started successfully kubectl get pods -n payment-service ## Expected: 2 pods in Running state echo "✅ Pod Security Admission layer configured" ```
### The default network problem By default, every pod in a Kubernetes cluster can reach every other pod on any port. A compromised `frontend` pod can connect directly to your `database` pod on port 5432. An attacker who escapes from the `payment-service` can probe every service in the cluster. **NetworkPolicy** is a Kubernetes firewall that controls pod-to-pod traffic. It works at the IP and port level. Without a NetworkPolicy, all traffic is allowed. With a NetworkPolicy, all traffic not explicitly allowed is denied. > 📌 **Remember:** NetworkPolicy requires a CNI (Container Network Interface) plugin that supports it. Flannel does not. Calico, Cilium, and Weave all do. Kind uses Kindnet by default — install Calico for NetworkPolicy support in this capstone. ```bash ## Install Calico CNI for NetworkPolicy support in kind kubectl apply -f https://raw.githubusercontent.com/projectcalico/calico/v3.27.0/manifests/calico.yaml ## Wait for Calico to be ready kubectl rollout status daemonset/calico-node -n kube-system --timeout=120s echo "✅ Calico CNI installed — NetworkPolicies now enforced" ``` ```yaml ## network-policies/default-deny-all.yaml ## The most important policy: deny everything unless explicitly allowed. ## Apply this first, then add allow policies for legitimate traffic. apiVersion: networking.k8s.io/v1 kind: NetworkPolicy metadata: name: default-deny-all namespace: payment-service spec: ## podSelector: {} means this policy applies to ALL pods in the namespace podSelector: {} ## policyTypes with both Ingress and Egress means deny all traffic in both directions policyTypes: - Ingress - Egress ``` ```yaml ## network-policies/allow-payment-ingress.yaml ## Allow external traffic to reach the payment service from the ingress controller apiVersion: networking.k8s.io/v1 kind: NetworkPolicy metadata: name: allow-payment-ingress namespace: payment-service spec: podSelector: matchLabels: app: payment-service policyTypes: - Ingress ingress: - from: ## Only allow traffic from the ingress-nginx namespace - namespaceSelector: matchLabels: kubernetes.io/metadata.name: ingress-nginx ## Within that namespace, only from pods labelled as the ingress controller podSelector: matchLabels: app.kubernetes.io/name: ingress-nginx ports: - protocol: TCP port: 8080 ``` ```yaml ## network-policies/allow-payment-egress.yaml ## Allow the payment service to make outbound connections it needs apiVersion: networking.k8s.io/v1 kind: NetworkPolicy metadata: name: allow-payment-egress namespace: payment-service spec: podSelector: matchLabels: app: payment-service policyTypes: - Egress egress: ## Allow DNS resolution — without this the service cannot resolve any hostnames - ports: - protocol: UDP port: 53 - protocol: TCP port: 53 ## Allow connecting to the database namespace only - to: - namespaceSelector: matchLabels: kubernetes.io/metadata.name: database podSelector: matchLabels: app: postgresql ports: - protocol: TCP port: 5432 ## Allow connecting to external payment processors (Razorpay, NPCI) ## In production: use CIDR ranges for specific IPs instead of any - ports: - protocol: TCP port: 443 ``` ```yaml ## network-policies/allow-monitoring-scrape.yaml ## Allow Prometheus to scrape metrics from the payment service apiVersion: networking.k8s.io/v1 kind: NetworkPolicy metadata: name: allow-monitoring-scrape namespace: payment-service spec: podSelector: matchLabels: app: payment-service policyTypes: - Ingress ingress: - from: - namespaceSelector: matchLabels: kubernetes.io/metadata.name: monitoring podSelector: matchLabels: app.kubernetes.io/name: prometheus ports: - protocol: TCP port: 9090 ``` ```bash ## Apply all network policies kubectl apply -f network-policies/ ## Test that the policies work ## Deploy a test pod in the payment-service namespace kubectl run test-pod \ --image=busybox:1.35 \ --restart=Never \ --namespace=payment-service \ -- sleep 3600 ## Should FAIL — default deny blocks inter-pod traffic kubectl exec -n payment-service test-pod -- \ wget -T 2 -q http://kubernetes.default.svc.cluster.local ## Expected: wget: download timed out ## Should SUCCEED — DNS is explicitly allowed kubectl exec -n payment-service test-pod -- \ nslookup kubernetes.default.svc.cluster.local ## Expected: successful DNS resolution echo "✅ Network Policy layer configured" ```
### Writing custom admission policies **OPA Gatekeeper** is an admission controller that validates Kubernetes resource requests against custom policies before they are created. It uses **Rego**, a policy language, to define rules. While Pod Security Admission handles common security profiles, Gatekeeper handles your organization-specific rules: * All production containers must have resource limits set * Container images must come from the approved registry only * All deployments must have at least 2 replicas * Labels for team and cost center are required on every namespace ```bash ## Install OPA Gatekeeper kubectl apply -f https://raw.githubusercontent.com/open-policy-agent/gatekeeper/v3.14.0/deploy/gatekeeper.yaml ## Wait for Gatekeeper to be ready kubectl rollout status deployment/gatekeeper-controller-manager \ -n gatekeeper-system --timeout=120s echo "✅ OPA Gatekeeper installed" ``` ```yaml ## gatekeeper/allowed-registries-template.yaml ## Define the policy template — this is the Rego logic ## A ConstraintTemplate defines what the policy checks. ## A Constraint (next file) configures where and how it applies. apiVersion: templates.gatekeeper.sh/v1 kind: ConstraintTemplate metadata: name: k8sallowedregistries annotations: description: "Require container images to come from approved registries only" spec: crd: spec: names: kind: K8sAllowedRegistries ## These are the parameters you configure in the Constraint validation: openAPIV3Schema: type: object properties: registries: type: array items: type: string targets: - target: admission.k8s.gatekeeper.sh rego: | package k8sallowedregistries violation[{"msg": msg}] { ## Get every container in the pod spec (including init containers) container := input.review.object.spec.containers[_] ## Check if the image starts with any of the allowed registries not starts_with_allowed(container.image) msg := sprintf( "Container '%v' uses image '%v' which is not from an approved registry. Allowed: %v", [container.name, container.image, input.parameters.registries] ) } ## Check initContainers too violation[{"msg": msg}] { container := input.review.object.spec.initContainers[_] not starts_with_allowed(container.image) msg := sprintf( "Init container '%v' uses image '%v' not from approved registry", [container.name, container.image] ) } starts_with_allowed(image) { ## startswith checks if image begins with one of the approved registry prefixes startswith(image, input.parameters.registries[_]) } ``` ```yaml ## gatekeeper/allowed-registries-constraint.yaml ## Apply the policy — specify allowed registries and which namespaces to check apiVersion: constraints.gatekeeper.sh/v1beta1 kind: K8sAllowedRegistries metadata: name: allowed-registries spec: ## enforcementAction: deny = block the resource if policy fails ## Use "warn" first when rolling out to see what would be blocked enforcementAction: deny match: kinds: - apiGroups: [""] kinds: ["Pod"] ## Apply to all namespaces except kube-system and gatekeeper-system excludedNamespaces: ["kube-system", "gatekeeper-system"] parameters: registries: - "ghcr.io/razorpay/" - "registry.k8s.io/" - "docker.io/library/" ``` ```yaml ## gatekeeper/require-resource-limits-template.yaml ## Policy: all containers must have CPU and memory limits apiVersion: templates.gatekeeper.sh/v1 kind: ConstraintTemplate metadata: name: k8srequirelimits annotations: description: "Require all containers to have CPU and memory resource limits" spec: crd: spec: names: kind: K8sRequireLimits targets: - target: admission.k8s.gatekeeper.sh rego: | package k8srequirelimits violation[{"msg": msg}] { container := input.review.object.spec.containers[_] ## Check if memory limit is missing not container.resources.limits.memory msg := sprintf( "Container '%v' is missing memory limit. Add resources.limits.memory.", [container.name] ) } violation[{"msg": msg}] { container := input.review.object.spec.containers[_] ## Check if CPU limit is missing not container.resources.limits.cpu msg := sprintf( "Container '%v' is missing CPU limit. Add resources.limits.cpu.", [container.name] ) } ``` ```yaml ## gatekeeper/require-resource-limits-constraint.yaml apiVersion: constraints.gatekeeper.sh/v1beta1 kind: K8sRequireLimits metadata: name: require-resource-limits spec: enforcementAction: deny match: kinds: - apiGroups: [""] kinds: ["Pod"] excludedNamespaces: ["kube-system", "gatekeeper-system"] ``` ```bash ## Apply all Gatekeeper policies kubectl apply -f gatekeeper/ ## Test the registry policy — should be rejected cat > /tmp/test-registry.yaml << 'EOF' apiVersion: v1 kind: Pod metadata: name: test-registry namespace: payment-service spec: containers: - name: app image: nginx:latest resources: limits: cpu: "100m" memory: "128Mi" EOF kubectl apply -f /tmp/test-registry.yaml ## Expected: ## Error from server (Forbidden): admission webhook "validation.gatekeeper.sh" denied: ## Container 'app' uses image 'nginx:latest' which is not from an approved registry. ## Allowed: ["ghcr.io/razorpay/", "registry.k8s.io/", "docker.io/library/"] ## Test the resource limits policy — should be rejected cat > /tmp/test-no-limits.yaml << 'EOF' apiVersion: v1 kind: Pod metadata: name: test-no-limits namespace: payment-service spec: containers: - name: app image: ghcr.io/razorpay/payment-service:latest EOF kubectl apply -f /tmp/test-no-limits.yaml ## Expected error about missing resource limits echo "✅ OPA Gatekeeper policies active" ```
A fintech startup running on Kubernetes had been in production for eight months. Their application was secure. Their cod...
What Kubernetes exposes by default Kubernetes is secure by design in some areas and dangerously permissive in others. Un...
Understanding Roles, ClusterRoles, and Bindings RBAC (Role-Based Access Control) is Kubernetes's permission system. It a...
What Pod Security Standards enforce Pod Security Admission (PSA) replaced the deprecated PodSecurityPolicy in Kubernetes...
The default network problem By default, every pod in a Kubernetes cluster can reach every other pod on any port. A compr...
Writing custom admission policies OPA Gatekeeper is an admission controller that validates Kubernetes resource requests ...
Detecting threats that slipped past admission The four previous layers are all preventive controls. They stop bad config...
Automated compliance auditing kube-bench is an open-source tool that checks Kubernetes cluster configuration against the...
...
...
Giving every new ServiceAccount the default ServiceAccount permissions. Kubernetes automatically creates a default Servi...
Layer Tool What It Controls Where Configured RBAC kubectl API access permissions ClusterRole / Role YAML Pod Security PS...
Aligns directly with DevOps, Site Reliability (SRE), and Platform Engineering job descriptions.