Kubernetes Production Readiness Challenge

This milestone deploys the sandbox app, its database and a ServiceMonitor so Prometheus scrapes it, and proves the system is healthy before anything is broken. A baseline you trust is what lets you tell a real signal from noise later.

Why a purpose-built app

The app has three properties that make each drill reproducible: it fails fast at startup if it cannot reach the database, its readiness probe checks the database, and it exposes request counters and a latency histogram. It also has two chaos switches, an ERROR_RATE variable and a /debug/leak endpoint, so you can create errors and memory growth on demand. Real services should not ship a leak endpoint.

Create the project layout and the app code:

app/package.json:

app/server.js:

app/Dockerfile:

Build and push the image. Replace the registry with yours:

Now the manifests. k8s/baseline.yaml is the single source of truth for the known good state:

Deploy it, and create a reset script that returns the lab to this known good state between scenarios:

Verify the baseline. All pods should be Running with 1/1 ready and zero restarts:

Send a minute of traffic, then check Prometheus. Open it with kubectl -n monitoring port-forward svc/monitoring-kube-prometheus-prometheus 9090 and run this query. You should see a non-zero rate:

Common Mistake

Skipping the baseline and discovering mid-scenario that Prometheus never scraped the app. If the query above is empty, check that the shop-api target is UP under Status then Targets in the Prometheus UI before going further.

A CrashLoopBackOff means a container starts, exits, and Kubernetes keeps restarting it with growing delays. It is usually a startup problem such as a missing variable, a bad config file or an unreachable dependency, so the evidence lives in the logs of the container that just died.

Inject

Clear the database host. The app fails fast when it cannot connect:

Within about a minute you will see new pods in Error then CrashLoopBackOff with a rising restart count. The old pods keep serving traffic because a rolling update never removes a healthy pod until its replacement is ready. That is why users notice nothing yet, and why a stuck rollout can hide for hours if you do not watch it.

Detect

The logs show startup failed: followed by a connection error. Notice what is missing: nothing says which variable is wrong. Exit code 1 means the application chose to exit, which separates this from exit code 137 (killed, Scenario 2).

Fix

Prevent

The fix for a confusing crash is a clear one. Turn on the app's startup validation, then repeat the injection and compare the log message:

You should now read FATAL: missing required environment variables: DB_HOST. The crash is self-diagnosing. Restore the host, and edit k8s/baseline.yaml so VALIDATE_ENV is "true" and reset.sh keeps it.

Common Mistake

Running kubectl logs without --previous on a crash-looping pod and concluding there are no logs. The current container is new and empty. Always read the previous one first.

An OOMKilled container did not crash on its own. The Linux kernel killed it because it exceeded its memory limit, and Kubernetes then restarted it. It is caused either by a limit set too low for real usage or by a leak that grows until it hits the ceiling.

Inject

Lower the limit to 128Mi, which is realistic for a small Node.js service, then make the app hold more memory than that. A limit of 32Mi would kill Node at startup, which is a different failure, so do not use it:

The first calls succeed. When the process crosses 128Mi the kernel kills it and the connection drops.

Detect

You should see Reason: OOMKilled and Exit Code: 137. Exit code 137 is 128 plus signal 9, so it always means the process was killed from outside. Now confirm it in Prometheus, where you can also see the shape of the problem over time:

The first graph climbs towards 128Mi and drops to near zero at the restart. The second turns to 1 for the killed container.

Fix

Size the limit from observed usage, not from a guess. Look at the working set during normal load, and take the high percentile rather than the average, then leave headroom:

Prevent

Add an alert so a restart caused by OOM pages someone instead of hiding in a restart counter. Save as k8s/oom-alert.yaml:

Common Mistake

Sizing limits from the average. If average usage is 100Mi but the 95th percentile is 180Mi, a 120Mi limit kills the pod during every traffic spike. Use a high percentile plus headroom, and treat repeated limit increases as a sign of a leak, not a fix.

Tip

The Vertical Pod Autoscaler in recommendation-only mode (updateMode: "Off") is a good next step. It suggests requests and limits from real usage without changing pods.

A bad rollout ships a version that cannot run. The most common causes are a wrong image tag, a broken image or a bad config, and the cleanest way to recover is to roll back first and investigate second.

Inject

Deploy a tag that does not exist, which is what a mistyped CI variable produces:

The rollout never completes. The command times out while old pods keep serving.

Detect

The new pod is in ErrImagePull then ImagePullBackOff, and events say the image could not be pulled. The Deployment shows fewer up-to-date replicas than desired.

Fix

The output should be your v1 image again. Rolling back is a one-line recovery. Fixing forward is fine later, once users are safe.

Prevent

Make the safe behaviour explicit instead of relying on defaults, and fail the pipeline before a missing image reaches the cluster. Add this to the shop-api Deployment spec in k8s/baseline.yaml:

maxUnavailable: 0 guarantees old pods stay until new ones are ready, and progressDeadlineSeconds marks a stalled rollout as failed so alerts and pipelines can react. Then add an image check to your CI deploy job:

Common Mistake

Treating a stuck rollout as success because users are fine. The old version is still running and the new one never shipped. Always check kubectl rollout status in the pipeline and fail the job when it times out.

Every PostgreSQL server has a hard limit on simultaneous connections. Each app replica keeps a pool of connections open, so total connections equal replicas multiplied by pool size. Scale out or enlarge the pool and the product can exceed the limit, after which new connections fail and requests return errors. In this lab the limit is 30 with 3 reserved for admins, so about 27 are usable.

Inject

Detect

You should see sorry, too many clients already in the database log, a connection count at the ceiling, and a share of 500 responses in the load tool's summary. In Prometheus, check the error rate:

Fix

Stop the damage first, then fix the structure. A connection pooler such as PgBouncer sits between the apps and the database, accepts many client connections and reuses a small number of real ones.

Deploy PgBouncer as k8s/pgbouncer.yaml. The environment variable names below are those of the edoburu/pgbouncer image, so check its documentation if you pick a different image:

Repeat the detection commands. The connection count to PostgreSQL should stay near the pooler's size of 10, and the load summary should show no 500 responses. Requests that cannot get a database connection now queue briefly instead of failing, so expect higher latency at the same load, which is the right trade.

Prevent

• Apply a sizing rule everywhere: replicas x pool size must stay below max_connections minus reserved connections. With a pooler, the rule applies to the pooler's server pool instead.
• Set a maximum on your HorizontalPodAutoscaler so scale-out cannot outgrow the database.
• Keep PgBouncer in the baseline by adding it to your reset flow.

Common Mistake

Raising max_connections on the database to make the errors stop. Each connection costs memory on the server, so a bigger limit just moves the cliff. Pool connections instead of multiplying them.

A cost spike is usually not an attack. It is a forgotten load test, a runaway scale-out or a workload with no resource requests, which the scheduler cannot account for and billing tools cannot attribute. If an autoscaler is installed, extra pods also mean extra nodes, which is where the money goes.

Inject

If your nodes are small, some pods will sit Pending. That is itself a signal, and on a cluster with an autoscaler it is the moment new nodes would launch.

Detect

In Prometheus, count containers that have no CPU request. These are the workloads that look free on paper but are not:

If you installed Kubecost in an earlier module, open Allocations grouped by namespace to see the same spike in money terms and a falling efficiency score.

Fix

Prevent

Two Kubernetes objects work together. A LimitRange gives every container default requests and limits, so nothing is created without them. A ResourceQuota caps the total a namespace can use. The quota needs the LimitRange, because a quota on requests rejects any pod that does not declare them. Save as k8s/guardrails.yaml:

The pod count now stops at the quota instead of reaching 30, and events show exceeded quota. For admission-time policy with friendlier error messages, a policy engine such as Kyverno is a good extension.

Common Mistake

Adding a ResourceQuota without a LimitRange. Every existing manifest that omits requests then fails to deploy with a confusing must specify requests error. Apply them together.

A NetworkPolicy that selects a pod makes that pod default-deny for the traffic types it lists. A deny-all policy with no rules therefore blocks everything, including DNS lookups on port 53, which is the most common way a well-meant security change breaks an application. Policies apply to new connections, so you must restart pods to see the full effect.

Inject

The old pods are replaced by new ones that cannot resolve the database name, so they exit and crash-loop.

Detect

The error name is the clue. EAI_AGAIN means name resolution is failing, which points to DNS or the network, not to your application. Also notice a second casualty: with ingress blocked, Prometheus can no longer scrape the pods, so your metrics go quiet exactly when you need them.

Fix

Then apply policies that are least-privilege but do not break the platform. Save as k8s/network-policies.yaml:

DNS should resolve, pods should be ready, and the shop-api target should be UP in Prometheus again.

Prevent

• Roll out policies to one namespace at a time, always starting with the DNS allow rule.
• After every policy change, run the same three checks: a DNS lookup from a pod, the readiness state, and the Prometheus target status.
• Remember that these policies now block everything else, including outbound internet traffic from the namespace. That is intended, but add explicit rules when a service legitimately needs it.

Common Mistake

Testing a new policy only against existing pods. Established connections survive, so everything looks fine until the next restart. Restart the workload after applying the policy to see what a fresh pod experiences.

An SLO (Service Level Objective) is an internal reliability target that guides engineering decisions. It is not a promise to customers, which is an SLA. When a service is inside its SLO the team ships features, and when it is outside, reliability work comes first.

Choose SLOs from data

Run the service for a while and look at real numbers before committing, because an SLO you cannot meet only teaches the team to ignore alerts. For this lab, use these:

The error budget is the allowed failure. Over 30 days there are 43,200 minutes, so a 99.9% target allows 43.2 minutes of full outage. "We have 20 minutes of budget left this month" is something a team can act on, while "we are at 99.93%" is not.

Why burn rate, not a threshold

A burn rate is how fast you consume the budget compared with the pace that would use it up exactly at the end of the period. A burn rate of 1 uses the whole budget in 30 days. A burn rate of 14.4 uses 2% of the monthly budget in one hour, which deserves a page. Alerting on burn rate tells you how urgent a problem is, not only that it exists. Using two windows, a long one to confirm the problem is real and a short one to confirm it is still happening, avoids pages for blips that have already ended.

Recording and alerting rules

Save as k8s/slo-rules.yaml:

Prove the alerts fire

Inject errors, send steady traffic for ten minutes, and watch the Alerts page in the Prometheus UI move from inactive to pending to firing:

Write a short runbook for each alert in docs/runbooks/: what the alert means, the first three commands to run, the likely causes and how you fixed it last time. An alert with no runbook only tells the on-call engineer that something is wrong.

Common Mistake

Setting a 99.9% SLO because it sounds professional, then discovering the service has never reached it. Measure for a few weeks first and set the target at what you can hold with normal effort. A 99.5% SLO you meet is worth more than a 99.9% SLO you constantly breach.

This milestone turns what you did into the two artifacts hiring managers and teammates actually read: a symptom-first playbook and a blameless postmortem for each scenario. Writing them forces you to separate what you knew from what you guessed.

The playbook

Start every investigation from the symptom and run the first command in the table:

For slow responses, two queries separate the usual suspects:

The postmortem template

Write one per scenario from your incident-log.md. Focus on systems and decisions, not on people:

Tip

If a postmortem's root cause is "someone made a mistake", keep asking why until you reach a missing guardrail. The guardrail is the real fix.

Reset the lab, then confirm every guardrail is in place and working.

Finally, complete this scorecard from your incident log. It is your portfolio summary:

Cleanup

Common Mistake

Leaving the sandbox running after the project. The database, the load pods and any autoscaled nodes keep billing. Delete the namespace and check that your cluster has scaled back down.