ResourceQuota ā Capping What Each Team Can Consume
Why This Matters in Shared Clusters
Without quotas on a multi-tenant cluster, one team running a memory leak can OOMKill every other team's pods by exhausting node memory. At Razorpay, where multiple teams share the same production cluster, ResourceQuotas are the enforcement boundary between teams.
ā DIAGRAM
+----------------------------------------------------+| mumbai-prod-cluster (total: 128 CPU, 512Gi RAM) |+----------------------------------------------------+ | | | v v v+---------------+ +---------------+ +---------------+| payments-team | | risk-team | | data-team || namespace | | namespace | | namespace || | | | | || Quota: | | Quota: | | Quota: || 32 CPU | | 16 CPU | | 64 CPU || 128Gi RAM | | 64Gi RAM | | 256Gi RAM || 100 pods | | 50 pods | | 200 pods |+---------------+ +---------------+ +---------------+A Complete ResourceQuota Manifest
YAML
1apiVersion: v12kind: ResourceQuota3metadata:4 name: payments-team-quota5 namespace: payments-prod6spec:7 hard:8 # āā Compute Resources āā9 requests.cpu: "32" # Total CPU requests across ALL pods in namespace10 requests.memory: 128Gi # Total memory requests across ALL pods11 limits.cpu: "64" # Total CPU limits across ALL pods12 limits.memory: 256Gi # Total memory limits across ALL pods13 # āā Object Count Limits āā14 pods: "100" # Max number of pods15 services: "20" # Max number of Services16 secrets: "50" # Max number of Secrets17 configmaps: "30" # Max number of ConfigMaps18 persistentvolumeclaims: "20" # Max number of PVCs19 # āā Service Type Limits āā20 services.loadbalancers: "3" # Limit expensive cloud load balancers21 services.nodeports: "0" # Block NodePort services entirelyWhat Happens When a Quota Is Hit
ā DIAGRAM
+------------------------------------------+| kubectl apply -f new-deployment.yaml | <- Team tries to deploy+------------------------------------------+ | v+------------------------------------------+| Admission Controller checks quota | <- Checks: used + requested <= hard+------------------------------------------+ | | v v+------------------+ +------------------------------------------+| UNDER QUOTA | | QUOTA EXCEEDED || Pod created OK | | Error: exceeded quota: payments-team- || | | quota, requested: requests.memory=4Gi, || | | used: requests.memory=126Gi, || | | limited: requests.memory=128Gi |+------------------+ +------------------------------------------+
Bash
1# What the error looks like in practice2kubectl apply -f new-deployment.yaml3# Error from server (Forbidden): pods "api-server-xyz" is forbidden:4# exceeded quota: payments-team-quota, requested: requests.memory=4Gi,5# used: requests.memory=126Gi, limited: requests.memory=128GiThe pod goes into Pending and won't schedule until other pods are removed or the quota is raised.
ResourceQuota Requires Resource Requests on Every Pod
š Remember: If a namespace has a ResourceQuota for CPU or memory, every pod in that namespace MUST specify resource requests and limits. Pods without requests will be rejected outright ā even if the namespace has plenty of quota headroom remaining.
YAML
1# This pod will be REJECTED if the namespace has a CPU/memory quota2containers:3 - name: api4 image: api:latest5 # Missing resources block -> rejected with "must specify requests" error6 7# This pod will be ACCEPTED8containers:9 - name: api10 image: api:latest11 resources:12 requests:13 cpu: "500m"14 memory: "512Mi"15 limits:16 cpu: "1"17 memory: "1Gi"Viewing Quota Usage
Bash
1# Summary view ā all quotas in a namespace2kubectl get quota -n payments-prod3 4# Detailed usage breakdown ā see used vs hard for every resource5kubectl describe quota payments-team-quota -n payments-prod6# Resource Used Hard7# -------- ---- ----8# limits.cpu 28 649# limits.memory 96Gi 256Gi10# persistentvolumeclaims 14 2011# pods 67 10012# requests.cpu 14 3213# requests.memory 48Gi 128Gi14# secrets 31 5015# services 12 2016# services.loadbalancers 2 317# services.nodeports 0 018 19# Check quota across all namespaces at once20kubectl get quota -AResourceQuota vs LimitRange ā How They Work Together
ā DIAGRAM
+------------------------------------------+| ResourceQuota | <- Namespace ceiling:| | "This namespace gets 32 CPU total"| Enforced at: admission time || Scope: entire namespace |+------------------------------------------+ | v+------------------------------------------+| LimitRange | <- Per-container guardrails:| | "Each container: 100m-4 CPU"| Enforced at: admission time || Scope: individual container |+------------------------------------------+Use both together. ResourceQuota without LimitRange means a single container can claim all 32 CPU in one pod. LimitRange without ResourceQuota means per-container limits are set but the namespace has no ceiling ā 1000 small pods could still exhaust the cluster.
Troubleshooting Common ResourceQuota Problems
| Problem | Symptom | Fix |
|---|---|---|
| New pods rejected with Forbidden | exceeded quota error on kubectl apply |
Check kubectl describe quota to see which resource is exhausted ā scale down unused deployments or request a quota increase |
| Pod rejected with "must specify requests" | failed quota even when under limits |
Namespace has ResourceQuota but pod spec has no resources block ā add explicit CPU and memory requests |
| Quota shows 0 used but pods exist | Old quota object not matching namespace | Quota metadata.namespace doesn't match the namespace you're deploying into ā verify with kubectl get quota -A |
| Services.nodeports: 0 but NodePort needed | Service creation blocked | Request a quota change or use an Ingress instead of NodePort ā NodePorts are blocked intentionally in multi-tenant clusters |
| Quota raised but pod still pending | Pod stuck after quota increase | The pod was rejected before the quota increase ā delete and reapply the pod after the quota is updated |
š” Tip: Set up a Prometheus alert when any namespace hits 80% of its quota. At 100%, new deployments silently fail ā which is very confusing during an incident when you're trying to scale up your pods to handle a traffic spike.
ā ļø Security: Use services.nodeports: "0" in production quotas to block NodePort services across all team namespaces. NodePorts expose services directly on the node's public IP ā on a shared cluster like Razorpay's, this bypasses all ingress-level authentication and rate limiting.š“ Common Mistake: Setting a quota only onrequests.cpuandrequests.memorybut not onlimits.cpuandlimits.memory. A developer can setrequests: 100mbutlimits: 64 CPUon a single container ā the quota passes (requests are under the cap), but the container can burst and consume the entire node.