Platform Engineering Foundations

When 10 teams share one Kubernetes cluster, you need isolation. Without it: * Team A's memory leak exhausts nodes, crashing Team B's pods * Team C accidentally deploys to Team D's namespace * A misconfigured pod in the payments namespace can reach the user-service database * One team's workload consumes all available CPU, starving everyone else Multi-tenancy is the practice of safely sharing a cluster across multiple teams or customers. ### The Two Models of Multi-Tenancy **Namespace per team** — Each team owns one or more namespaces. Policies, RBAC, quotas, and network policies enforce isolation between namespaces. Teams share the same Kubernetes control plane. This is the most common approach and what we build in this module. **Virtual control plane per team** — Each team gets their own Kubernetes API server, controller manager, and etcd. Complete isolation but much higher cost and complexity. Used when teams need cluster-admin level access or have strict compliance requirements. Tools like vCluster implement this pattern. For most organisations, namespace-per-team provides the right balance of isolation and operational simplicity. ### Namespace-Per-Team — The Building Blocks A namespace is the fundamental unit of isolation in Kubernetes. Everything else — RBAC, ResourceQuotas, NetworkPolicies, LimitRanges — is scoped to a namespace. ```bash ## Create team namespaces with meaningful labels kubectl create namespace payments-production kubectl create namespace orders-production kubectl create namespace user-service-production ## Label namespaces for policy targeting kubectl label namespace payments-production \ team=payments \ environment=production \ tier=critical kubectl label namespace orders-production \ team=orders \ environment=production \ tier=standard ``` ### ResourceQuotas — Preventing Noisy Neighbours A noisy neighbour is a workload that consumes disproportionate cluster resources, degrading performance for everyone else. ResourceQuotas enforce hard limits per namespace. ```yaml ## payments-quota.yaml apiVersion: v1 kind: ResourceQuota metadata: name: payments-team-quota namespace: payments-production spec: hard: ## Compute — payments team gets guaranteed resources requests.cpu: "20" limits.cpu: "40" requests.memory: "40Gi" limits.memory: "80Gi" ## Storage requests.storage: "500Gi" persistentvolumeclaims: "20" ## Object counts — prevent runaway deployments pods: "100" services: "20" configmaps: "50" secrets: "50" ## Prevent creating LoadBalancer services without approval services.loadbalancers: "2" ``` ```bash kubectl apply -f payments-quota.yaml ## Check quota usage for a namespace kubectl describe resourcequota -n payments-production ## Output: ## Name: payments-team-quota ## Namespace: payments-production ## Resource Used Hard ## -------- ---- ---- ## limits.cpu 8 40 ## limits.memory 16Gi 80Gi ## pods 12 100 ## requests.cpu 4 20 ``` ### LimitRanges — Default Resource Requests ResourceQuotas enforce namespace-level limits. LimitRanges enforce container-level defaults. Without LimitRanges, developers who forget to set resource requests deploy pods with no resource requests — they consume unlimited resources and are invisible to the scheduler for capacity planning. ```yaml ## default-limitrange.yaml ## Apply to every team namespace apiVersion: v1 kind: LimitRange metadata: name: default-limits namespace: payments-production spec: limits: - type: Container ## Default values injected if developer does not specify default: cpu: "500m" memory: "512Mi" defaultRequest: cpu: "100m" memory: "128Mi" ## Absolute bounds — even explicit values cannot exceed these min: cpu: "50m" memory: "64Mi" max: cpu: "8" memory: "16Gi" ``` > 📌 **Remember:** LimitRanges only apply to new pods. Existing pods are not affected when you apply a LimitRange. To enforce limits on existing workloads, you need to trigger a rolling restart. ### Network Policies — Traffic Isolation By default, every pod in a Kubernetes cluster can reach every other pod. In a multi-tenant cluster this means a compromised pod in one team's namespace can make requests to any other team's database. Network Policies implement microsegmentation — explicit allow rules, with everything else denied. ```yaml ## default-deny-all.yaml ## Apply this to every team namespace first ## Then add explicit allow rules for what needs to communicate apiVersion: networking.k8s.io/v1 kind: NetworkPolicy metadata: name: default-deny-all namespace: payments-production spec: podSelector: {} ## applies to all pods in this namespace policyTypes: - Ingress - Egress --- ## allow-dns.yaml ## DNS must be allowed for any pod to function apiVersion: networking.k8s.io/v1 kind: NetworkPolicy metadata: name: allow-dns-egress namespace: payments-production spec: podSelector: {} policyTypes: - Egress egress: - ports: - port: 53 protocol: UDP - port: 53 protocol: TCP --- ## allow-within-namespace.yaml ## Pods in the same namespace can talk to each other apiVersion: networking.k8s.io/v1 kind: NetworkPolicy metadata: name: allow-same-namespace namespace: payments-production spec: podSelector: {} policyTypes: - Ingress ingress: - from: - podSelector: {} ## any pod in the same namespace --- ## allow-from-orders.yaml ## Explicitly allow orders-service to call payments-service apiVersion: networking.k8s.io/v1 kind: NetworkPolicy metadata: name: allow-from-orders namespace: payments-production spec: podSelector: matchLabels: app: payment-api policyTypes: - Ingress ingress: - from: - namespaceSelector: matchLabels: team: orders podSelector: matchLabels: app: order-service ports: - protocol: TCP port: 8080 ``` > ⚠️ **Security:** Network Policies require a CNI plugin that supports them — Calico, Cilium, or Weave Net. The default Kubernetes networking (Flannel) does not enforce Network Policies. If your CNI does not support them, NetworkPolicy resources are silently ignored. ---