Harden a Kubernetes Cluster End to End

Harden a kind cluster with RBAC, Pod Security Admission, NetworkPolicies, Gatekeeper, Falco and kube-bench, and prove each layer blocks attacks.

Architecture Overview & Problem Statement

This project hardens a local Kubernetes cluster in six layers: RBAC for who can do what, Pod Security Admission for what pods may look like, NetworkPolicies for who can talk to whom, OPA Gatekeeper for your own admission rules, Falco for runtime detection, and kube-bench for a CIS benchmark report. You prove each layer works by trying to break it and watching it refuse.

Kubernetes does what you ask and refuses nothing you have not told it to refuse. The goal is to make the safe path the only path, so a rushed change cannot ship a privileged pod, open network paths or an over-permissioned account.

flowchart LR A[kubectl apply] --> B[RBAC allowed?] B --> C[Pod Security Admission] C --> D[Gatekeeper policies] D --> E[Pod runs] E --> F[NetworkPolicy limits traffic] E --> G[Falco watches behaviour] H[kube-bench] -.audits.-> E

Prevent, restrict, detect, audit

Layer	Tool	Type	Question it answers
1	RBAC	Prevent	Who may call which API?
2	Pod Security Admission	Prevent	Is this pod spec safe?
3	NetworkPolicy	Restrict	Which pods may connect?
4	Gatekeeper	Prevent	Does this follow our own rules?
5	Falco	Detect	Is something suspicious happening now?
6	kube-bench	Audit	How does the cluster compare to the CIS benchmark?

What you need first

Docker, kubectl and Helm installed on Linux, macOS or WSL2
About 6 GB of free memory for a two-node kind cluster
Basic Kubernetes knowledge: pods, deployments, services, namespaces

Security
This runs on a disposable local cluster. Some tests deliberately try forbidden things. Do not run them against a shared or production cluster.

Tip
Each layer is applied to a single namespace called hardened, and Gatekeeper rules are scoped to it. That keeps system components in kube-system and tool namespaces such as falco untouched, which avoids the most common way hardening breaks a cluster.

Milestone 1: Create the Cluster with a Network Policy Engine

This milestone creates a two-node kind cluster that can enforce NetworkPolicies. kind's default network plugin has limited policy support, so you start the cluster without it and install Calico, which enforces policies fully. The plugin has to be chosen when the cluster is created, so do this before anything else.

Install kind with a verified checksum

Bash

curl -Lo kind https://github.com/kubernetes-sigs/kind/releases/download/v0.33.0/kind-linux-amd64
echo "aee6151561422756b764a4ae28e7f44cda5af5a9eead3cc9985112b1de8d8e0d  kind" | sha256sum -c -
chmod +x kind && sudo mv kind /usr/local/bin/kind

On macOS, use brew install kind.

Create the cluster

YAML

# kind-config.yaml
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
networking:
  disableDefaultCNI: true
  podSubnet: 192.168.0.0/16
nodes:
  - role: control-plane
  - role: worker

Bash

kind create cluster --name hardening --config kind-config.yaml
kubectl get nodes

Expected result: both nodes show NotReady. That is correct, because no network plugin exists yet.

Install Calico

Bash

kubectl apply -f https://raw.githubusercontent.com/projectcalico/calico/v3.33.0/manifests/calico.yaml
kubectl -n kube-system rollout status daemonset/calico-node --timeout=300s
kubectl get nodes

Expected result: both nodes become Ready. The pod range 192.168.0.0/16 matches Calico's default, so no extra configuration is needed.

Create the working namespace

Bash

kubectl create namespace hardened

Common Mistake
Installing Calico on a kind cluster that was created with the default network plugin. The two fight over pod networking and policies silently do nothing. If you skipped disableDefaultCNI, delete the cluster with kind delete cluster --name hardening and recreate it.

Milestone 2: RBAC with Least Privilege

RBAC (role-based access control) decides which identity may perform which action on which resource. The rule to follow is least privilege: give each identity only what it needs. A compromised pod inherits its service account's permissions, so an over-permissioned account turns one bug into a cluster-wide incident.

Create an app account and a read-only developer role

The app does not call the Kubernetes API at all, so it gets a service account with no token mounted. Developers get read access to workloads but no access to secrets and no ability to open a shell in a pod, because pods/exec is effectively a shell in production.

YAML

# rbac.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
  name: app
  namespace: hardened
automountServiceAccountToken: false
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: developer-readonly
  namespace: hardened
rules:
  - apiGroups: [""]
    resources: ["pods", "pods/log", "services", "events", "configmaps"]
    verbs: ["get", "list", "watch"]
  - apiGroups: ["apps"]
    resources: ["deployments", "replicasets"]
    verbs: ["get", "list", "watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: developer-readonly
  namespace: hardened
subjects:
  - kind: Group
    name: developers
    apiGroup: rbac.authorization.k8s.io
roleRef:
  kind: Role
  name: developer-readonly
  apiGroup: rbac.authorization.k8s.io

Bash

kubectl apply -f rbac.yaml

Test the permissions

kubectl auth can-i with --as impersonates an identity, so you can test without creating real users.

Bash

D="--as=dev@example.com --as-group=developers -n hardened"
kubectl auth can-i list pods $D
kubectl auth can-i get secrets $D
kubectl auth can-i create pods/exec $D
kubectl auth can-i get secrets -n hardened --as=system:serviceaccount:hardened:app

Expected result: yes for listing pods, then no for secrets, no for exec, and no for the app account reading secrets.

Audit cluster-admin bindings

Bash

kubectl get clusterrolebindings -o custom-columns='NAME:.metadata.name,ROLE:.roleRef.name,SUBJECTS:.subjects[*].name' | grep cluster-admin

Expected result: only system bindings such as cluster-admin and system:masters. Any other entry deserves a question.

Common Mistake
Giving a role pods/exec because developers "need to debug". Offer read access to logs by default and grant exec as a temporary, audited exception.

Milestone 3: Pod Security Admission

Pod Security Admission (PSA) is built into Kubernetes and checks every pod against one of three standards, set by labels on the namespace. privileged allows everything, baseline blocks the most dangerous settings such as privileged containers and host namespaces, and restricted adds requirements like running as non-root, dropping all capabilities and a default seccomp profile.

Enforce the restricted standard

Bash

kubectl label namespace hardened \
  pod-security.kubernetes.io/enforce=restricted \
  pod-security.kubernetes.io/enforce-version=latest \
  pod-security.kubernetes.io/warn=restricted \
  pod-security.kubernetes.io/audit=restricted

Watch it refuse a privileged pod

YAML

# bad-pod.yaml
apiVersion: v1
kind: Pod
metadata:
  name: bad
  namespace: hardened
spec:
  containers:
    - name: bad
      image: busybox:1.37
      command: ["sleep", "3600"]
      securityContext:
        privileged: true

Bash

kubectl apply -f bad-pod.yaml

Expected result: an error that starts with violates PodSecurity "restricted:latest" and lists the privileged container among the reasons.

Deploy a compliant app

The sample app is podinfo, a small public web service. The security context below is what the restricted standard requires. Note that readOnlyRootFilesystem is good practice but is not required by the restricted standard.

YAML

# app.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: podinfo
  namespace: hardened
spec:
  replicas: 2
  selector:
    matchLabels:
      app: podinfo
  template:
    metadata:
      labels:
        app: podinfo
    spec:
      serviceAccountName: app
      automountServiceAccountToken: false
      securityContext:
        runAsNonRoot: true
        runAsUser: 10001
        seccompProfile:
          type: RuntimeDefault
      containers:
        - name: podinfo
          image: ghcr.io/stefanprodan/podinfo:6.15.0
          ports:
            - containerPort: 9898
          securityContext:
            allowPrivilegeEscalation: false
            capabilities:
              drop: ["ALL"]
          resources:
            requests:
              cpu: 50m
              memory: 64Mi
            limits:
              cpu: 200m
              memory: 128Mi
---
apiVersion: v1
kind: Service
metadata:
  name: podinfo
  namespace: hardened
spec:
  selector:
    app: podinfo
  ports:
    - port: 9898
      targetPort: 9898

Bash

kubectl apply -f app.yaml
kubectl -n hardened rollout status deployment/podinfo --timeout=180s

Expected result: the deployment becomes ready with two pods.

Tip
Roll out a new standard with warn and audit labels first. They report violations without blocking, so you can find workloads that would break before you switch to enforce.

Common Mistake
Labelling kube-system or tool namespaces as restricted. Network plugins and node agents legitimately need privileges. Apply restricted to application namespaces only.

Milestone 4: NetworkPolicies

By default every pod can reach every other pod. A NetworkPolicy is a firewall rule for pods: once any policy selects a pod, traffic not explicitly allowed is dropped. The method is to deny everything first, then allow only the flows you can name.

Default deny, plus three narrow allows

YAML

# netpol.yaml
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny
  namespace: hardened
spec:
  podSelector: {}
  policyTypes: [Ingress, Egress]
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-dns
  namespace: hardened
spec:
  podSelector: {}
  policyTypes: [Egress]
  egress:
    - to:
        - namespaceSelector:
            matchLabels:
              kubernetes.io/metadata.name: kube-system
          podSelector:
            matchLabels:
              k8s-app: kube-dns
      ports:
        - protocol: UDP
          port: 53
        - protocol: TCP
          port: 53
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: app-ingress
  namespace: hardened
spec:
  podSelector:
    matchLabels:
      app: podinfo
  policyTypes: [Ingress]
  ingress:
    - from:
        - podSelector:
            matchLabels:
              access: frontend
      ports:
        - protocol: TCP
          port: 9898
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: frontend-egress
  namespace: hardened
spec:
  podSelector:
    matchLabels:
      access: frontend
  policyTypes: [Egress]
  egress:
    - to:
        - podSelector:
            matchLabels:
              app: podinfo
      ports:
        - protocol: TCP
          port: 9898

Bash

kubectl apply -f netpol.yaml

Test with two clients

Both test pods follow the restricted standard and carry resource limits, so they pass the earlier layers. Only one carries the access: frontend label.

YAML

# clients.yaml
apiVersion: v1
kind: Pod
metadata:
  name: frontend
  namespace: hardened
  labels:
    access: frontend
spec:
  automountServiceAccountToken: false
  securityContext:
    runAsNonRoot: true
    runAsUser: 10001
    seccompProfile:
      type: RuntimeDefault
  containers:
    - name: c
      image: busybox:1.37
      command: ["sleep", "3600"]
      securityContext:
        allowPrivilegeEscalation: false
        capabilities:
          drop: ["ALL"]
      resources:
        limits:
          cpu: 50m
          memory: 32Mi
---
apiVersion: v1
kind: Pod
metadata:
  name: stranger
  namespace: hardened
spec:
  automountServiceAccountToken: false
  securityContext:
    runAsNonRoot: true
    runAsUser: 10001
    seccompProfile:
      type: RuntimeDefault
  containers:
    - name: c
      image: busybox:1.37
      command: ["sleep", "3600"]
      securityContext:
        allowPrivilegeEscalation: false
        capabilities:
          drop: ["ALL"]
      resources:
        limits:
          cpu: 50m
          memory: 32Mi

Bash

kubectl apply -f clients.yaml
kubectl -n hardened wait --for=condition=ready pod/frontend pod/stranger --timeout=120s
kubectl -n hardened exec frontend -- wget -qO- -T 3 http://podinfo:9898/healthz
kubectl -n hardened exec stranger -- wget -qO- -T 3 http://podinfo:9898/healthz

Expected result: the frontend pod prints a short JSON health response, and the stranger pod fails with a timeout. The timeout is the policy dropping the packets.

Common Mistake
Applying a default-deny egress policy and forgetting DNS. Every name lookup fails and the app looks broken. The allow-dns policy above is the first thing to add after a default deny.

Tip
A policy is only enforced if your network plugin supports it. That is why Milestone 1 installed Calico. If the stranger pod can reach the app, check that the plugin pods are running.

Milestone 5: Custom Admission Rules with Gatekeeper

Pod Security Admission covers common safety settings. OPA Gatekeeper covers your own rules, written in the Rego policy language. You define a template (the logic), then a constraint (where and with what parameters it applies). Here you enforce two rules in the hardened namespace: images must come from approved sources, and every container must have CPU and memory limits.

Install Gatekeeper

Bash

kubectl apply -f https://raw.githubusercontent.com/open-policy-agent/gatekeeper/v3.23.1/deploy/gatekeeper.yaml
kubectl -n gatekeeper-system rollout status deployment/gatekeeper-controller-manager --timeout=180s

Define the templates

YAML

# gatekeeper-templates.yaml
apiVersion: templates.gatekeeper.sh/v1
kind: ConstraintTemplate
metadata:
  name: k8sallowedrepos
spec:
  crd:
    spec:
      names:
        kind: K8sAllowedRepos
      validation:
        openAPIV3Schema:
          type: object
          properties:
            repos:
              type: array
              items:
                type: string
  targets:
    - target: admission.k8s.gatekeeper.sh
      rego: |
        package k8sallowedrepos
        import rego.v1
 
        violation contains {"msg": msg} if {
          some c in input.review.object.spec.containers
          not image_allowed(c.image)
          msg := sprintf("container %v uses image %v, which is not from an approved source", [c.name, c.image])
        }
 
        image_allowed(img) if {
          some prefix in input.parameters.repos
          startswith(img, prefix)
        }
---
apiVersion: templates.gatekeeper.sh/v1
kind: ConstraintTemplate
metadata:
  name: k8srequirelimits
spec:
  crd:
    spec:
      names:
        kind: K8sRequireLimits
  targets:
    - target: admission.k8s.gatekeeper.sh
      rego: |
        package k8srequirelimits
        import rego.v1
 
        violation contains {"msg": msg} if {
          some c in input.review.object.spec.containers
          not c.resources.limits.memory
          msg := sprintf("container %v has no memory limit", [c.name])
        }
 
        violation contains {"msg": msg} if {
          some c in input.review.object.spec.containers
          not c.resources.limits.cpu
          msg := sprintf("container %v has no CPU limit", [c.name])
        }

Bash

kubectl apply -f gatekeeper-templates.yaml
sleep 15

The pause gives Gatekeeper time to create the new constraint types from the templates.

Apply the constraints to one namespace

YAML

# gatekeeper-constraints.yaml
apiVersion: constraints.gatekeeper.sh/v1beta1
kind: K8sAllowedRepos
metadata:
  name: approved-sources
spec:
  enforcementAction: deny
  match:
    kinds:
      - apiGroups: [""]
        kinds: ["Pod"]
    namespaces: ["hardened"]
  parameters:
    repos:
      - "ghcr.io/stefanprodan/"
      - "busybox:"
---
apiVersion: constraints.gatekeeper.sh/v1beta1
kind: K8sRequireLimits
metadata:
  name: require-limits
spec:
  enforcementAction: deny
  match:
    kinds:
      - apiGroups: [""]
        kinds: ["Pod"]
    namespaces: ["hardened"]

Bash

kubectl apply -f gatekeeper-constraints.yaml

Test both rules

The first pod meets the restricted standard but uses an image that is not on the approved list. The second uses an approved image but has no limits.

Bash

cat > untrusted.yaml << 'EOF'
apiVersion: v1
kind: Pod
metadata:
  name: untrusted
  namespace: hardened
spec:
  securityContext:
    runAsNonRoot: true
    runAsUser: 10001
    seccompProfile:
      type: RuntimeDefault
  containers:
    - name: c
      image: nginxinc/nginx-unprivileged:1.27
      securityContext:
        allowPrivilegeEscalation: false
        capabilities:
          drop: ["ALL"]
      resources:
        limits:
          cpu: 100m
          memory: 64Mi
EOF
cat > nolimits.yaml << 'EOF'
apiVersion: v1
kind: Pod
metadata:
  name: nolimits
  namespace: hardened
spec:
  securityContext:
    runAsNonRoot: true
    runAsUser: 10001
    seccompProfile:
      type: RuntimeDefault
  containers:
    - name: c
      image: busybox:1.37
      command: ["sleep", "3600"]
      securityContext:
        allowPrivilegeEscalation: false
        capabilities:
          drop: ["ALL"]
EOF
kubectl apply -f untrusted.yaml
kubectl apply -f nolimits.yaml

Expected result: both are denied by the admission webhook, with the messages from the Rego rules. The running podinfo pods are unaffected, because admission only checks new objects.

Common Mistake
Applying a constraint to every namespace. Policies that block system images or require limits will stop kube-system pods from restarting. Always scope with namespaces or excludedNamespaces, and try new rules with enforcementAction: warn first.

Milestone 6: Runtime Detection with Falco and a CIS Audit

The earlier layers are preventive. Falco is detective: it watches system calls from running containers and raises an alert when behaviour looks wrong, such as a shell starting inside a container that should never run one. kube-bench then audits the cluster configuration against the CIS Kubernetes Benchmark.

Security
Falco needs kernel features that some environments lack, such as certain WSL2 and older Docker Desktop kernels. If the Falco pods crash on start, check their logs, and treat this layer as optional on that machine rather than lowering other settings.

Install Falco with a custom rule

The rule uses Falco's built-in spawned_process macro and shell_binaries list, so you do not redefine them.

YAML

# falco-values.yaml
driver:
  kind: modern_ebpf
tty: true
customRules:
  hardened-rules.yaml: |-
    - rule: Shell in hardened namespace
      desc: A shell was started in a container in the hardened namespace.
      condition: >
        spawned_process and container and proc.name in (shell_binaries)
        and k8s.ns.name = "hardened"
      output: >
        Shell started in hardened namespace
        (ns=%k8s.ns.name pod=%k8s.pod.name container=%container.name
        user=%user.name cmd=%proc.cmdline)
      priority: WARNING
      tags: [hardened, shell]

Bash

helm repo add falcosecurity https://falcosecurity.github.io/charts
helm repo update
helm search repo falcosecurity/falco --versions | head -3
helm upgrade --install falco falcosecurity/falco -n falco --create-namespace -f falco-values.yaml
kubectl -n falco rollout status daemonset/falco --timeout=300s

Pin the chart by adding --version with the number the search command shows, so later installs match.

Trigger and read the alert

Bash

kubectl -n hardened exec deploy/podinfo -- sh -c 'echo hello'
kubectl -n falco logs -l app.kubernetes.io/name=falco -c falco --tail=100 | grep "Shell started in hardened"

Expected result: a log line that names the namespace, the pod and the command. A real deployment sends such alerts to a chat or paging tool, because an alert nobody sees is just a log.

Run kube-bench

Bash

kubectl apply -f https://raw.githubusercontent.com/aquasecurity/kube-bench/v0.16.0/job.yaml
kubectl wait --for=condition=complete job/kube-bench --timeout=180s
kubectl logs job/kube-bench > kube-bench.txt
grep -E '^\[(FAIL|WARN)\]' kube-bench.txt | head -20
grep -A6 '== Summary' kube-bench.txt
kubectl delete job kube-bench

Expected result: a list of failed and warned checks and a summary count for each area. On a kind cluster many results reflect how kind configures its components, so use this run to learn to read the report: each check has an ID, a status and a remediation, and WARN means a person must decide, not that the check passed.

Status	Meaning
PASS	The setting matches the benchmark
FAIL	The setting does not match and should be fixed or justified
WARN	Needs manual review, depends on your environment
INFO	Informational only

Common Mistake
Ignoring every WARN and chasing only FAIL. Many warnings are real gaps. Review each one and record a decision.

Validation & Testing

This script proves every layer still holds. Run it after any change to the cluster, and keep it in your repository as evidence.

Bash

#!/usr/bin/env bash
# verify.sh
set -u
pass=0; fail=0
ok()   { echo "PASS: $1"; pass=$((pass+1)); }
bad()  { echo "FAIL: $1"; fail=$((fail+1)); }
 
D="--as=dev@example.com --as-group=developers -n hardened"
[ "$(kubectl auth can-i list pods $D)" = "yes" ] && ok "developers can list pods" || bad "developers can list pods"
[ "$(kubectl auth can-i get secrets $D)" = "no" ] && ok "developers cannot read secrets" || bad "developers cannot read secrets"
[ "$(kubectl auth can-i create pods/exec $D)" = "no" ] && ok "developers cannot exec" || bad "developers cannot exec"
 
kubectl apply -f bad-pod.yaml 2>&1 | grep -q 'violates PodSecurity' && ok "PSA blocks privileged pod" || bad "PSA blocks privileged pod"
kubectl apply -f untrusted.yaml 2>&1 | grep -qi 'denied' && ok "Gatekeeper blocks untrusted image" || bad "Gatekeeper blocks untrusted image"
kubectl apply -f nolimits.yaml 2>&1 | grep -qi 'denied' && ok "Gatekeeper requires limits" || bad "Gatekeeper requires limits"
 
kubectl -n hardened exec frontend -- wget -qO- -T 3 http://podinfo:9898/healthz >/dev/null 2>&1 && ok "frontend reaches app" || bad "frontend reaches app"
kubectl -n hardened exec stranger -- wget -qO- -T 3 http://podinfo:9898/healthz >/dev/null 2>&1 && bad "stranger blocked" || ok "stranger blocked"
 
echo "passed=${pass} failed=${fail}"
[ "${fail}" -eq 0 ]

Bash

chmod +x verify.sh && ./verify.sh

Expected result: eight PASS lines and failed=0. If a line fails, the matching milestone tells you which layer to inspect.

Checklist

The nodes are Ready and Calico pods are running.
A privileged pod is refused by Pod Security Admission.
An unapproved image and a pod without limits are refused by Gatekeeper.
The stranger pod times out and the frontend pod connects.
Falco logged the shell, if your machine supports it.
You have a saved kube-bench report with one fixed or documented item.

Reflection questions

Which of these layers would have stopped a pod with hostPID: true, and which would only have reported it?
Why scope Gatekeeper rules to a namespace, and what breaks if you do not?
Why is a default-deny policy followed immediately by a DNS allow?
What is the difference between a preventive and a detective control, and why do you need both?

Portfolio deliverable

Your README should contain the layer diagram, the output of verify.sh, one real Gatekeeper denial message, and a short note on one kube-bench finding and how you handled it.

Tip
Stretch goals: enable API audit logging with an audit policy, route Falco alerts to a webhook, and write one more Gatekeeper rule such as requiring a team label.

Found a mistake, an outdated step or a broken link?

Harden a Kubernetes Cluster End to End

Blueprint Walkthrough

Architecture Overview & Problem Statement

Prevent, restrict, detect, audit

What you need first

Milestone 1: Create the Cluster with a Network Policy Engine

Install kind with a verified checksum

Create the cluster

Install Calico

Create the working namespace

Milestone 2: RBAC with Least Privilege

Create an app account and a read-only developer role

Test the permissions

Audit cluster-admin bindings

Milestone 3: Pod Security Admission

Enforce the restricted standard

Watch it refuse a privileged pod

Deploy a compliant app

Milestone 4: NetworkPolicies

Default deny, plus three narrow allows

Test with two clients

Milestone 5: Custom Admission Rules with Gatekeeper

Install Gatekeeper

Define the templates

Apply the constraints to one namespace

Test both rules

Milestone 6: Runtime Detection with Falco and a CIS Audit

Install Falco with a custom rule

Trigger and read the alert

Run kube-bench

Validation & Testing

Checklist

Reflection questions

Portfolio deliverable