Production Troubleshooting Interview Questions

Confirm the impact, tell people, and stabilise the system before you hunt for the root cause. Panicking and diving straight into logs is the most common mistake.

A good first five minutes looks like this:

1. Confirm it is real and measure the scope. Look at the dashboard: who is affected, which feature, since when, and how bad (error rate, latency, traffic drop).
2. Declare the incident. Open a channel, name an incident commander, and tell stakeholders that you are looking at it.
3. Ask what changed. Recent deploys, config changes, feature flags, certificate expiry, traffic spikes and cloud provider events explain most outages.
4. Pick the fastest safe mitigation. Rollback, failover, scale up, disable the feature, or block bad traffic.
5. Investigate the cause after users are safe.

Interviewers like this answer because it shows you separate "stop the bleeding" from "find the cause".

Mitigation makes the user impact stop quickly; a root cause fix removes the underlying defect so it does not return. During an incident you mitigate first, and you fix the root cause afterwards.

Examples of mitigation: rolling back a release, failing over to another region, restarting a stuck service, adding capacity, turning off a feature flag. These do not explain why the problem happened.

Root cause work (finding the bug, fixing the config, adding a missing index) can take hours or days and belongs in the postmortem and follow-up tasks.

The trap is spending 40 minutes proving the cause while customers are still down, when a 2-minute rollback would have ended the impact. The opposite trap is mitigating and then never doing the follow-up, so the incident repeats.

Search the kernel log for OOM killer messages. When the system runs out of memory, the kernel kills a process and writes the details to the kernel log, not to your application log.

The message shows which process died, its memory use, and often the memory state of the whole machine.

Two points often missed:

• In containers, the limit may be the cgroup limit rather than the host's memory. The host can have plenty of free RAM and the container still gets OOM-killed.
• The killed process is not always the one leaking. The kernel picks a victim by its oom_score, which usually favours large processes.

Fixes: find and fix the leak, raise the memory limit, set sensible limits for other services, or add memory.

Find what is using the space, free some of it safely, then fix the reason it filled up.

The usual culprits are logs, old Docker images and container logs, temp files, core dumps and package caches.

Safe ways to free space:

• Compress or delete old rotated logs.
• journalctl --vacuum-size=500M to shrink the systemd journal.
• docker system df then prune unused images (carefully, on a known host).
• To empty a log that a running process still writes to, use truncate -s 0 file instead of rm, because deleting an open file does not free the space until the process closes it.

Then prevent a repeat: set up log rotation, add a disk usage alert at 80%, and find out why the growth happened.

The filesystem has probably run out of inodes, not bytes. An inode is the record the filesystem keeps for each file. Millions of tiny files can use up all inodes while using very little space.

Typical sources are session files, mail queues, cache directories, build artefacts, and a runaway process creating tiny files.

Fix: delete the files you identified (using find ... -delete rather than rm *, which can fail with "argument list too long"). Long term, clean up automatically, move the data to a better storage design, or recreate the filesystem with more inodes if the workload really needs it.

Interviewers use this question to check that you do not stop at df -h.

The machine has run out of physical memory and is moving memory pages to disk, which is thousands of times slower than RAM. When it constantly swaps pages in and out, this is called thrashing.

Having some swap used is not a problem by itself. Continuous si and so activity is the danger sign.

Steps: find the memory-hungry process, reduce its load or restart it, then fix the underlying cause (a leak, a limit set too high, or too little RAM for the workload). Adding swap only delays the problem and slows the machine down further.

On Kubernetes nodes, swap is normally disabled so the scheduler's memory accounting stays accurate.

A 502 means the proxy could not get a valid response from the upstream (the backend app). The proxy itself is working; the thing behind it is not.

Check in this order:

1. Is the backend running and listening on the expected port? (ss -tlnp, systemctl status)
2. Can the proxy host reach it? Try curl -v http://upstream:port/ from the proxy.
3. Read the proxy error log: "connect() failed (111: Connection refused)" means nothing listens; "upstream prematurely closed connection" means the app crashed or closed mid-response.
4. Look for restarts or crashes in the app log, and recent deploys (during a rolling deploy, instances may be dropped).
5. Check for a keepalive mismatch: if the backend closes idle connections sooner than the proxy expects, you get sporadic 502s.

They point to different places, so the status code is your first clue.

The first sentence of your answer to a "what does 5xx mean" question should name who generated it: 500 usually comes from the app, 502 to 504 usually from a proxy or load balancer in front of it. Always check which component generated the response, because a load balancer can return its own 503 without ever touching the app.

This points to a DNS problem: the name is not resolving to the right address from where you test. Check what the machine actually resolves.

Common causes:

• A wrong or missing DNS record, or a record pointing at an old IP.
• A stale cache (local, resolver or application) holding an old answer until the TTL expires.
• An entry in /etc/hosts overriding DNS.
• The machine uses a resolver that cannot see your private zone.
• A search domain adding a suffix you did not expect.

Note that dig bypasses /etc/hosts and the OS cache, so also test with getent to see what applications actually get. Many programs also cache DNS themselves (the JVM does, by default under certain settings).

"Connection refused" means the host answered and said no, while "timed out" means nothing answered at all. The two point to different causes.

• Connection refused: the packet reached the machine, but nothing is listening on that port (service down, wrong port, bound to another interface), or a firewall actively rejected it with a reset.
• Connection timed out: the packets were silently dropped or never arrived. Think firewall or security group that drops, a routing problem, a wrong IP, or a host that is down.

In practice: refused means "go look at the service on that host". Timed out means "look at the network path and the firewall rules". This one distinction saves a lot of time, so say it in your first sentence.

Test from the outside in: name, path, port, listener, firewall, application.

1. Resolve the name: dig +short host.
2. Test the port from the client: nc -vz host 8080 or curl -v.
3. On the server, check it listens: ss -tlnp | grep 8080. If it shows 127.0.0.1:8080, the app only accepts local connections and must bind to 0.0.0.0.
4. Host firewall: iptables -S or nft list ruleset, plus tools like ufw or firewalld.
5. Network firewalls: cloud security groups, network ACLs (these are stateless, so both directions need rules), and route tables.
6. The application: is it healthy and not stuck?

Cloud flow logs (like VPC Flow Logs) show REJECT entries for blocked traffic. The usual cause of "works locally, fails remotely" is binding to localhost.

Another process is already listening on that port, or the previous instance has not released it yet. Find which one.

Possible reasons:

• An old copy of the same service is still running (a failed stop or a duplicate start).
• A different application uses the same port.
• The port is in TIME_WAIT after a restart. Servers normally avoid this with the SO_REUSEADDR option.
• A container or Kubernetes hostPort conflicts with something on the node.

Fix: stop or reconfigure the conflicting process, or change the port. Do not just kill -9 something you have not identified, because it may be an important service.

Read the container's logs and exit code first. Most failures are visible there.

Common causes:

• The main process finished or crashed (a container lives only as long as its main process).
• Missing environment variable, config file or secret.
• Wrong CMD or ENTRYPOINT, or the file is not executable.
• exec format error: the image was built for another CPU architecture (for example arm64 on an amd64 host).
• Permission problems, because the container runs as a non-root user.

If the application runs in the background and exits (like a daemonised process), the container stops. Run it in the foreground.

The most common causes are unused images and containers, build cache, and container logs that grow without limit.

The default json-file log driver has no size limit, so one chatty container can fill the disk. Set limits:

Be careful with docker volume prune and docker system prune -a: they can delete data or images you still need. Check what will be removed first, and avoid running them blindly on production hosts.

CrashLoopBackOff means the container keeps starting and crashing, and Kubernetes waits longer between each restart (the "backoff"). The restart is not the bug. Find out why the container exits.

Read the exit code: 1 usually means an application error, 137 means killed (often OOM), 143 means it received SIGTERM.

Frequent causes: a missing config map, secret or environment variable, a wrong command or image, an application bug on startup, a failing liveness probe that kills a healthy but slow app, or an unreachable dependency at startup.

If there is no log output, use kubectl debug or run the same image with a shell override to test.

Kubernetes docs: Debug Pods

Pending means the scheduler has not placed the pod on a node yet, or the pod cannot start. The reason is almost always in the events.

Typical messages and causes:

• Insufficient cpu or memory: no node has enough free resources for the pod's requests. Add nodes, lower requests, or let the cluster autoscaler add capacity.
• Taints not tolerated: nodes are tainted and the pod lacks a matching toleration.
• Node selector or affinity that matches no node.
• Unbound PersistentVolumeClaim: the volume is not provisioned, or is in another zone than the available nodes.
• ResourceQuota exceeded in the namespace.

Remember that the scheduler uses requests, not actual usage. A cluster can look half empty and still be unable to schedule a pod with large requests.

The node cannot download the container image. The event message tells you which of a few causes it is.

Check:

• Image name or tag typo, or the tag does not exist.
• Private registry without credentials: the pod needs an imagePullSecret (or the node role needs registry permissions).
• Registry rate limits (for example Docker Hub anonymous pulls).
• Network access from the node to the registry (egress rules, proxy, DNS).
• Wrong CPU architecture for the node.

Test from the node itself with the container runtime's pull command to separate cluster problems from node problems. As a good practice, use specific version tags or digests instead of latest, and mirror critical images in your own registry.

You are calling the API faster than its rate limit allows. Slow down, spread out your calls, and reduce the number of calls.

Short term: retry with exponential backoff and jitter (most SDKs have this built in, so make sure it is enabled and tuned).

Medium term:

• Reduce calls: cache results, batch requests, use list or describe calls with filters instead of one call per item.
• Find the noisy caller: logs or CloudTrail show which application or script makes the most calls. A forgotten polling loop is common.
• Spread work over time instead of running everything on the hour.
• Request a quota increase if the load is legitimate.

Throttling is often shared across an account and region, so one misbehaving job can break other teams' automation. Add rate limiting on your side so you do not depend on the provider to protect you.

If a server's clock is wrong, anything that depends on time breaks in confusing ways. Typical examples are TLS certificate errors, expired tokens, failed request signatures and mixed-up logs.

Examples:

• A TLS certificate looks "not yet valid" or "expired" because the clock is off.
• JWT or OAuth tokens are rejected as expired or issued in the future.
• Cloud API signatures fail (AWS rejects requests when the time differs by more than a few minutes).
• Log order across servers makes no sense, so incident timelines are wrong.
• Kerberos authentication and distributed databases misbehave.

Fix: make sure an NTP service (chrony or systemd-timesyncd) is running and can reach a time server. In clouds, use the provider's time service. VMs that were paused or restored from a snapshot can have large jumps.

Several layers can block traffic, so check each one in turn instead of guessing: security groups, network ACLs, host firewalls and routes.

Checklist for a cloud network:

1. Security group on the destination (inbound rule for the port and source) and the source (outbound rule).
2. Network ACLs on the subnets. They are stateless, so you need rules for both the request and the return traffic.
3. Host firewall on the server: iptables -S, nft list ruleset, ufw status, firewall-cmd --list-all.
4. Route tables: is there a route to the destination, and a route back?
5. Flow logs (such as VPC Flow Logs): a REJECT entry shows traffic was blocked at the cloud layer.

A timeout suggests a drop by a firewall, while an immediate refusal suggests the port is closed on the host.

A zombie process is a child process that has finished but still has an entry in the process table because its parent has not collected its exit status. It shows as Z or <defunct> in ps.

A few zombies are harmless: they use no CPU or memory, only a process table slot. A large number can exhaust the process ID table and prevent new processes from starting.

You cannot kill a zombie, because it is already dead. Instead, fix the parent: make the parent call wait(), or restart or kill the parent so that init (PID 1) adopts and reaps the zombies.

In containers, the main process often runs as PID 1 and does not reap children. Use a small init such as tini (docker run --init, or shareProcessNamespace and an init in Kubernetes) so zombies are cleaned up.

On a shared branch like main, use git revert, which creates a new commit that undoes the bad one. Do not rewrite shared history.

For a merge commit, specify which parent to keep: git revert -m 1 <merge-sha>.

Then redeploy so production gets the reverting commit, or roll back the deployment to the previous artefact, which is often faster.

Why not git reset --hard and force push? That rewrites history that others have already pulled, which breaks their local branches and can drop other people's commits. reset is fine on your own local, unpushed work.

For a quick production fix, rolling back the deployment is often the fastest action, and the revert commit follows to keep the repository consistent.

Assign clear roles, use one channel, and share short regular updates that separate facts from guesses. Confusion costs more time than the technical problem.

Common roles:

• Incident commander: coordinates, decides, and does not debug.
• Operations lead: hands-on investigation and fixes.
• Communications lead: updates stakeholders, customers and the status page.
• Scribe: records the timeline (what was tried, when).

Good practice:

• One shared channel or call, and one place for the timeline.
• Regular updates (every 15 to 30 minutes), even when the update is "no change, still investigating".
• State impact, current status, next step and next update time.
• Avoid blaming and speculation in public channels.
• Keep a clear handover when people rotate.

The wrong approach is many people changing things in parallel with no coordination, which makes it impossible to know what helped or hurt.

Google SRE Book: Managing Incidents

A good postmortem explains what happened, why the system allowed it, and what will change, without blaming individuals. Its purpose is learning and prevention.

Typical sections:

• Summary and impact: what failed, who was affected, how long, and any data loss.
• Timeline: detection, response, mitigation, resolution, with times.
• Root cause and contributing factors: usually several, such as a missing test, a risky default, poor alerting or unclear runbooks.
• What went well and what went badly during detection and response.
• Action items: specific, with an owner and a due date, and tracked until done.

Blameless does not mean nobody is accountable. It means you assume people acted sensibly with the information they had, and you ask "how did the system let this happen?" rather than "who did it?" People who fear blame hide information, and then you cannot learn.

The common weakness is a list of vague actions like "be more careful". Good actions change the system.

Google SRE Book: Postmortem Culture

Walk the request path from the outside in, and test each layer until one fails. This turns a vague complaint into a specific broken component.

Practical steps:

First confirm the problem is not only on the user's side (try from another network). Then check whether DNS is right, whether the load balancer sees healthy targets, whether the app answers directly, and whether the app's dependencies (database, cache) respond. The first layer that fails is where you dig deeper.

Work out whether the problem is the network path, the SSH service, or the whole machine being starved of resources.

1. Network: does ping or nc -vz host 22 work? A timeout points to a firewall, security group or routing issue. "Connection refused" means the host is up but nothing listens on port 22.
2. Cloud console: check instance status checks, CPU, network and disk graphs. A machine at 100% CPU or swapping heavily can take minutes to answer a login.
3. Out-of-band access: use the provider's serial console, session manager or a configured bastion.
4. Common causes: memory exhaustion and swap thrashing, disk full (sshd cannot write), sshd crashed, too many sessions, a changed security group, or an expired key.

If you must recover by rebooting, capture evidence first when possible (a snapshot, console output). The common wrong answer is "just reboot it". That clears the symptom and destroys the evidence you need for the cause.

Find which process uses the CPU, then find out what type of CPU time it is, then connect it to what changed.

Read the CPU line carefully:

• us (user) high: your application code is busy. Profile it, or look for an infinite loop or a traffic spike.
• sy (system) high: lots of system calls, context switching or network handling. Try strace -c -p <pid>.
• wa (iowait) high: the CPU is waiting for disk, so the real bottleneck is storage.
• st (steal) high: on a VM, the hypervisor is giving your CPU time to someone else. Move or resize the instance.

Then correlate: did it start with a deploy, a cron job, a traffic increase or a new customer query? Mitigation is usually restarting the process, scaling out, or throttling the heavy workload. The CPU number alone never tells you the cause.

Look at "available" memory and per-process RSS over time, not at "free" memory. Linux deliberately fills spare RAM with the page cache, so low "free" is normal and healthy.

A real leak looks like this: one process's RSS grows steadily with no plateau, it drops only when the process restarts, and it correlates with traffic or runtime. Page cache growth, by contrast, shrinks automatically when applications need memory.

To dig deeper: check smem or /proc/<pid>/smaps for the breakdown, use a language-specific heap profiler (heap dumps for Java, pprof for Go, tracemalloc for Python), and compare a fresh process against an old one.

A short-term mitigation is a scheduled restart or a memory limit that restarts the process. That is a bandage; the leak still needs a fix.

A 504 means the proxy waited for the backend longer than its timeout. Raising the timeout is rarely the right first move because it hides slowness and ties up more connections.

Steps:

1. Measure backend latency directly (access logs with upstream response time, or traces). Is the backend slow for everyone or for specific requests?
2. Check what the backend waits on: slow queries, locked rows, a slow third-party API, exhausted connection pool, or saturated CPU.
3. Check timeouts across the chain. The client, CDN, load balancer, proxy and app each have one, and the inner ones should be shorter than the outer ones.
4. Look for a retry storm that adds load.

When a raise is acceptable: the request is genuinely long (a report, an export). Even then, a better design is to return quickly and process the work asynchronously.

A common trap: the load balancer idle timeout (for example 60 seconds) is lower than the app's real processing time.

Roll back by default when the change is the likely cause and rollback is safe and fast. Fix forward only when the fix is small, well understood, and rollback is risky or impossible.

Roll back when:

• The problem started right after the deploy.
• Users are affected now.
• Rollback is tested and quick.

Consider fixing forward when:

• The fix is a one-line change you are very sure about and a new release is fast.
• Rolling back is unsafe, for example a database migration already changed the schema or data in a way the old version cannot read.
• Several releases are bundled and rolling back drops other critical fixes.

The trap is "let's just push a quick fix", which stacks an untested change on a broken system. Restore service first, then investigate in a calm state.

Treat the deploy as the prime suspect, reduce impact first, then prove what changed. "Green health checks" only prove the app answers the health endpoint, not that real traffic works.

1. Limit impact: roll back or stop the rollout (in a canary, send traffic back to the old version).
2. Compare before and after: error rate by endpoint, status codes, log messages that are new since the deploy.
3. Check what the release changed: code, dependencies, config, environment variables, feature flags, database schema.
4. Look at dependencies: did the new version call a service differently (more load, new queries, a missing permission)?
5. Check environment differences: a secret or config that exists in staging but not in production.

Prevention: use canary or progressive delivery with automatic rollback tied to real error and latency metrics, and make health checks meaningful.

"Nothing changed" is never true: something outside the repository changed. Look for the first real error, then check what the pipeline depends on that you do not control.

Steps:

1. Read the first error in the log, not the last one. Later errors are often side effects.
2. Re-run once to check if it is flaky.
3. Compare with the last green run: runner image, tool versions, base image tags, dependency versions.
4. Check external dependencies: package registry outages, Docker Hub rate limits, expired tokens or certificates, a rotated secret, a changed API.
5. Check the runner: disk full, out of memory, a new runner version, a changed network rule.
6. Look for time-based issues: tests that depend on the date, expiring test certificates.

Common culprits: an unpinned dependency (latest) that released a breaking version, and an expired credential. Prevention: pin versions, use lock files, cache or mirror dependencies.

Measure first: find which stage or step takes the time, then fix the biggest one. Optimising a step that takes 20 seconds out of 20 minutes is wasted effort.

Common fixes:

• Queue time: the job waits for a free runner. Add runners or autoscale them.
• Dependency install: cache package directories keyed by lock file hash.
• Docker builds: order the Dockerfile so rarely changing layers come first, use layer caching, and use multi-stage builds.
• Tests: run them in parallel, split by timing, and run only affected tests on pull requests.
• Independent stages: run them in parallel instead of in sequence.
• Heavy checks: move slow scans or end-to-end tests to a nightly or post-merge run.

A trap: caching without a correct key, which makes builds fast but wrong. Always make caches invalidate when the inputs change.

Revert the configuration to the last known good version first, then work on why a bad config could reach production.

Response: find the exact change (config in version control makes this a one-line diff), revert it, restart or reload only what is needed, and confirm recovery with real traffic.

Prevention:

• Keep config in version control with review.
• Validate it before applying (schema checks, nginx -t, linting, dry runs).
• Roll it out gradually: one instance, then a small percentage, then everyone.
• Separate config deploys from code deploys so you know which one broke things.
• Give config changes the same CI checks and an automatic rollback.

A frequent root cause is that config is edited by hand on servers. Treat config as code.

Renew and deploy the certificate on every place that serves it, then verify from outside. Afterwards, find out why the renewal did not happen.

Check the expiry:

Fix: issue a new certificate, install it (including the intermediate chain), and reload the service, load balancer or ingress. Remember every endpoint that has its own copy: CDN, load balancers, internal services, mail servers.

Why it failed: a broken renewal job (cert-manager or certbot), failed ACME challenge, changed DNS, expired API credentials, or a manual process nobody owned.

Prevention: automate renewal, and monitor the certificate expiry date with alerts at 30, 14 and 7 days. Alerting on the renewal job alone misses cases where the job silently never ran.

The process has reached its limit of open file descriptors. Sockets and pipes count as files too, so this affects busy network services.

Decide whether it is a limit that is too low or a leak. If the count grows steadily and never falls, something is not closing connections or files. If it is stable near a low limit like 1024, raise it.

Raising the limit: for systemd services set LimitNOFILE=65535 in the unit file (the shell's ulimit does not apply to services). For containers, set --ulimit nofile.

Raising the limit on a leak only postpones the failure, so also fix the leak (unclosed HTTP clients, missing close() calls).

Exit code 137 means the process was killed by signal 9 (SIGKILL), because exit codes above 128 are 128 plus the signal number (128 + 9 = 137). Most often it means the container ran out of memory.

If this prints true, the container hit its memory limit and the kernel killed it. In Kubernetes, kubectl describe pod shows Reason: OOMKilled under the last state.

Other reasons for 137: someone ran docker kill or kill -9, or an orchestrator killed the container after it ignored a stop request for too long.

Related codes worth knowing: 143 is SIGTERM (a normal graceful stop), 139 is a segmentation fault (SIGSEGV), and 1 is a generic application error.

Fix OOM by raising the limit or reducing memory use. Check that the runtime respects the container limit (older JVMs read host memory).

The container used more memory than its limit, so the kernel killed it. You either need to give it more memory or reduce what it uses.

Steps:

1. Look at memory usage over time (metrics): is it a steady climb (leak) or a spike on certain requests (large payloads, batch jobs)?
2. Compare real usage against requests and limits.
3. Raise the limit if the workload truly needs it, and set the request close to normal usage so scheduling is honest.
4. For JVM or similar runtimes, make sure heap settings fit inside the container limit, leaving room for non-heap memory.

The memory limit counts everything in the container, including page cache from files and native memory. Setting the limit exactly at the average usage guarantees kills at the peaks.

Check whether the Service has endpoints. A Service forwards traffic only to pods that match its selector and are ready.

If the endpoints list is empty:

• The Service selector does not match the pod labels (a typo is very common).
• The pods are not Ready (a failing readiness probe removes them from endpoints).

If endpoints exist but traffic fails:

• port and targetPort do not match the container's real port.
• The app listens only on 127.0.0.1 inside the pod.
• A NetworkPolicy blocks the traffic.

Test from inside the cluster with a debug pod: kubectl run tmp --rm -it --image=busybox -- wget -qO- http://service:port. Test the pod IP directly to separate Service issues from app issues.

The new pods are failing their readiness probe (or crashing), so the Deployment will not remove old pods. That is the rolling update protecting you.

Check the probe settings:

• Wrong path, port or scheme (HTTP vs HTTPS).
• The application needs longer to start than initialDelaySeconds allows. Use a startupProbe for slow starters.
• The readiness probe depends on a downstream service, so one outage makes every pod unready.
• A bad config or missing secret in the new version.

Mitigation: kubectl rollout undo deployment/<name> returns to the previous ReplicaSet. Because of maxUnavailable and maxSurge, a stuck rollout normally keeps most old pods serving, so users are often still fine while you debug.

The kubelet evicts pods when a node runs low on a resource such as memory, disk or inodes, to protect the node. Eviction is the node defending itself.

Common cases: the node's ephemeral storage is full (container logs, emptyDir volumes, image layers) or memory is low.

Which pods go first: pods using more than their requests, and those with the lowest QoS class (BestEffort, then Burstable). Pods with requests equal to limits (Guaranteed) go last.

Fix: set realistic requests and limits, set ephemeral-storage limits, rotate logs, and clean up the node. Evicted pods remain as records, so delete them with kubectl delete pods --field-selector=status.phase=Failed.

The trap is deleting evicted pods without fixing the pressure: new ones will be evicted again.

A 404 usually means the ingress controller found no matching rule, while a 503 means a rule matched but there is no healthy backend to send traffic to.

For a 404:

• The Host header or path does not match any Ingress rule.
• The ingressClassName points to another controller, so your controller ignores it.
• The request hit the controller's default backend.

For a 503:

• The Service has no ready endpoints (selector mismatch, failing readiness probes).
• The Service port in the Ingress is wrong.

Also confirm that TLS secrets exist in the same namespace as the Ingress. Test by curling the controller with the right Host header: curl -H "Host: app.example.com" http://<controller-ip>/.

The database reached its maximum number of client connections. Find out who is holding them before you raise any limit.

For PostgreSQL:

Look for many connections in idle or idle in transaction, and for one service or host using most of them.

Common causes: connection leaks in the app, too many app instances each with a large pool (instances times pool size can exceed the DB limit), long transactions, or a retry storm after an earlier failure.

Fixes: correct the pool sizes, fix leaks, set timeouts for idle transactions, and add a pooler such as PgBouncer.

The trap is just raising max_connections. Each connection uses memory and the database often gets slower with more of them, so a pooler is usually the better answer.

Check from the application's point of view, layer by layer, and compare with what changed.

1. Is the database up? Check its status and logs, and try connecting from another host.
2. DNS: does the hostname resolve to the correct IP? After a failover the endpoint may have changed, and apps can cache the old IP.
3. Network: nc -vz db-host 5432 from the app host. Check security groups, firewalls, routes.
4. Authentication: was the password rotated, a user removed, or an IAM token expired?
5. TLS: did the database start requiring TLS, or did a certificate change?
6. Capacity: "too many connections", a full disk on the DB, or the DB in recovery mode.
7. Changes: deploys, config, maintenance windows, version upgrades.

The error text matters: "timeout" suggests network, "connection refused" suggests nothing is listening, "authentication failed" suggests credentials, and "too many connections" suggests capacity.

Replication lag means the replica applies changes more slowly than the primary produces them. The result is stale reads, and a risk of data loss if the primary fails and you promote the replica.

Common causes:

• A burst of writes or a heavy batch job on the primary.
• A long-running query on the replica that blocks applying changes (conflicts between queries and replay).
• Slow disks or low CPU on the replica.
• A single-threaded apply process that cannot keep up with a multi-threaded primary.
• Network bandwidth or latency problems between the nodes.

Check lag with pg_stat_replication (PostgreSQL) or SHOW REPLICA STATUS (MySQL). Cloud databases expose a lag metric you should alert on.

Mitigation: stop heavy queries on the replica, resize it, throttle the writes, or route lag-sensitive reads to the primary. Applications that need "read your own writes" must not read from a lagging replica right after a write.

Yes. If the application's connection pool is exhausted, requests wait in line for a free connection before they ever reach the database, so the database looks quiet while users wait.

Signs:

• Errors like "timeout waiting for connection from pool" or "unable to acquire connection".
• Latency rises while database CPU and query time stay normal.
• Pool metrics show active connections at the maximum and many waiters.

Causes:

• Connection leaks: code that borrows a connection and never returns it (missing close in an error path).
• Slow queries or long transactions that hold connections for too long.
• A pool that is too small for the traffic.
• A slow external call made while holding a connection.

Fix the leak or the slow query first. Making the pool bigger is the least effective fix, and it can overload the database. Always expose pool metrics (active, idle, waiting) so you can see this directly.

Check how Redis is configured to behave at its memory limit, then find what is using the memory. The behaviour depends on maxmemory and the eviction policy.

• With noeviction, Redis refuses writes when full and clients see "OOM command not allowed". That is correct behaviour for data you cannot lose.
• With policies like allkeys-lru, Redis deletes keys to make room, so you may see a falling cache hit rate instead of errors.

Common causes: keys without a TTL, a few huge keys (large lists or hashes), and a growth in traffic.

Fixes: add TTLs, remove or split big keys, raise memory, or add shards. Choose the policy on purpose: a cache can evict, but a session store or queue usually must not.

Consumer lag means the consumers read more slowly than producers write. First find out whether consumers are slow, stuck, or constantly rebalancing.

Look at the lag per partition:

• Lag on every partition: consumers are too slow overall. Check processing time per message, a slow downstream call, or too few consumers.
• Lag on one partition: a partition skew (a hot key) or one stuck consumer.
• Lag with no consumer assigned: the group lost members or keeps rebalancing (look for frequent rebalance logs, slow processing exceeding max.poll.interval.ms).

Fixes: speed up processing, add consumers (up to the number of partitions), fix the slow dependency, tune batch size, or add partitions with care (it changes key ordering). A poison message that fails repeatedly can also freeze one partition, so use a retry topic and dead-letter topic.

Find out whether messages are arriving faster than they are processed, or whether processing has stopped. The age of the oldest message tells you more than the queue depth.

Check:

1. Are consumers running and healthy? A crashed worker fleet means zero processing.
2. Is processing slow? A slow dependency or a heavier message type.
3. Is there a poison message that fails, returns to the queue and blocks others?
4. Did the producer rate change? A new feature, a retry storm or a batch job.

Actions: scale workers if the bottleneck is processing power, fix or bypass the slow dependency, move failing messages to a dead-letter queue after a few attempts, and throttle or pause producers if needed.

Watch for the trap in visibility timeouts: if a message takes longer than the timeout, it becomes visible again and is processed twice, which makes the backlog worse. Make handlers idempotent.

First check whether the targets really are broken, or whether the health check itself is wrong. A bad health check can take down a healthy fleet.

Check:

1. Health check configuration: path, port, protocol and expected status code. A recent change here is a prime suspect.
2. Network access: the security group or firewall must allow the load balancer's health check source to reach the target port.
3. Direct test: curl the health check URL from a host in the same network.
4. Application behaviour: is it slow to start, or too slow to answer within the check timeout?
5. Dependencies in the health check: if /health queries the database and the database blips, every target goes unhealthy at once, and then the load balancer has nothing to send traffic to.

Good practice: keep the health endpoint cheap and focused on "can this instance serve traffic". Many load balancers have a fail-open mode that sends traffic anyway when all targets are unhealthy, so know how yours behaves.

The load balancer is balancing connections or requests in a way that does not match real work, so look at connection behaviour and balancing rules.

Common causes:

• Long-lived connections (gRPC, HTTP/2, WebSockets) stay pinned to the instance they first connected to. New instances after scale-out get nothing.
• Sticky sessions that tie heavy users to one instance.
• Round robin treating heavy and light requests equally.
• Hot keys or partitions, where one key gets most of the traffic.
• Health check flapping that moves load around.

Fixes: use least outstanding requests balancing, set a maximum connection age so clients reconnect and rebalance, use client-side or L7 load balancing for gRPC, reduce reliance on sticky sessions, and spread hot keys.

Check per-instance metrics, not only the fleet average, because the average hides the imbalance.

Check the whole chain: is the metric reported, is the scaling rule triggered, and is there capacity to add? Any broken link stops scaling.

1. The metric: is it being collected and correct? In Kubernetes the HPA needs the metrics server, and CPU percentage requires resources.requests.cpu to be set on the pods.
2. The rule: threshold, evaluation period and cooldown. A long cooldown blocks a second scale-out.
3. Limits: the maximum replica or instance count may already be reached.
4. Capacity: cloud quotas, no available instance type in the zone, or pods Pending because the cluster autoscaler cannot add nodes.
5. Permissions or errors in the scaler itself.

Also check whether the scaled-out instances were created but failed health checks, which looks like "it did not scale" from the outside.

AWS runs two checks: the system status check tests AWS's underlying hardware and network, and the instance status check tests your operating system inside the instance.

• System check failed: the problem is on the AWS host. Stopping and starting the instance (not just rebooting) moves it to different hardware. You can also set an auto-recovery alarm.
• Instance check failed: the OS is likely at fault: a kernel panic, a full disk, a broken network configuration, a bad fstab entry or exhausted memory.

For instance check failures, use console output and the instance screenshot to see boot messages. If the OS will not boot, stop the instance, detach its root volume, attach it to a rescue instance, fix the files, and attach it back.

Take a snapshot before risky repair steps. Prevent repeats with an alarm on StatusCheckFailed and with automation that replaces failed instances instead of repairing them by hand.

Find the exact denied API call and the identity that made it, then work out which policy layer denied it. CloudTrail records both.

Steps:

1. Search CloudTrail for the error event: it shows the action, the resource and the principal (role or user).
2. Check that the app is using the role you expect (for example not an old instance profile or a different assumed role).
3. Review policies in the order AWS evaluates them. A request is denied if there is an explicit deny anywhere, or no allow. Check the identity policy, permission boundaries, service control policies (SCPs) from the organisation, resource policies (bucket or key policies) and session policies.
4. Use the IAM policy simulator to test the action.
5. For encrypted resources, check the KMS key policy too.

The trap is only editing the role's policy. The block is often an SCP, a resource policy or a KMS key policy in another place.

A 403 on S3 means the request was understood but denied by some layer. Check from the identity side and the bucket side.

Likely causes:

• The caller's IAM policy does not allow the action, or it was changed.
• The bucket policy denies it (including conditions on IP, VPC endpoint, or required encryption).
• Block Public Access settings prevent public access you expected.
• KMS permissions: objects encrypted with a customer key need kms:Decrypt or kms:GenerateDataKey on that key.
• Object ownership: objects uploaded by another account may not be readable by the bucket owner.
• A VPC endpoint policy restricts the bucket.
• Expired credentials or a wrong account or region.

Use CloudTrail or S3 server access logs to see the reason, and test with aws s3api head-object using the same credentials. Note that S3 may return 403 instead of 404 when the caller lacks s3:ListBucket, which hides whether an object exists.

Do not panic and do not edit the state by hand. Find out why it failed, check whether another run still holds the lock, then re-run plan.

If the apply failed halfway, some resources were created and recorded in state, others not. Terraform saves what it did, so run terraform plan again: it shows the remaining difference. Fix the cause (a permissions error, a quota, an invalid value) and apply again.

If you get "Error acquiring the state lock":

• First check whether another apply or a CI job is genuinely still running.
• Only if you are sure nobody is running, use terraform force-unlock <LOCK_ID>.

If the lock stays after a crashed run, force-unlock is the correct fix. If a real apply is still running and you unlock it, two applies can corrupt state.

Always use a remote backend with locking, and keep versioning on the state storage so you can restore an earlier version.

Terraform docs: State locking

Drift means the real infrastructure no longer matches what Terraform's state and code describe, usually because someone changed it manually in the console or another tool changed it.

Decide for each difference:

• The manual change was a mistake: run apply to put the infrastructure back to the code.
• The manual change was needed: update the Terraform code to match (and, if needed, terraform apply -refresh-only to accept the new real values in state).
• A new resource was created manually: import it with terraform import or an import block, or delete it.

Prevention: limit console write access, run scheduled plan jobs that alert on differences, and make the pipeline the only way to change infrastructure. Also note that some attributes change by design (tags added by a cloud service), which you can handle with ignore_changes.

The servers have drifted apart in configuration, packages or data. Compare the good and the bad server, then fix the way servers are built so drift cannot happen.

Compare:

• Package versions: dpkg -l or rpm -qa.
• Config files: use diff or checksums on key files, or ansible --check --diff to show what would change.
• Kernel parameters: sysctl -a, plus limits and mounted filesystems.
• Running services and environment variables.
• Hardware or instance type, and software versions of the runtime.

After you find the difference, ask how it got there: a manual hotfix, a failed run of the config tool, an automatic update that happened on only some hosts, or an old base image.

The best fix is immutable infrastructure: replace the server from a known image instead of repairing it, so every server is identical and any manual change disappears at the next replacement.

OutOfSync means the live cluster state differs from what Git says it should be, while Degraded means the resources exist but one or more is unhealthy (for example a Deployment whose pods are not ready). They are separate: an app can be synced and degraded.

Causes of OutOfSync: someone changed a resource manually with kubectl, a controller mutates a field (like replicas set by an HPA, or an injected sidecar), or a sync failed because of an invalid manifest or an immutable field.

Fixes: sync again, fix the manifest in Git, or tell Argo to ignore fields that other controllers manage (ignoreDifferences). With selfHeal on, Argo reverts manual changes, which is usually what you want in production.

For Degraded, debug the unhealthy workload like any Kubernetes problem.

Look at the release status and history, then roll back to the last working revision. Then find out why the upgrade failed.

Common causes:

• Rendered manifests are invalid: test with helm template or helm lint and --dry-run --debug.
• Pods never became ready, so the upgrade timed out (the real problem is in the workload, so debug the pods).
• A failed hook job.
• An immutable field change (like a Service clusterIP or a StatefulSet volume claim template).
• The release is stuck in pending-upgrade because an earlier run was interrupted.

Prevention: use helm upgrade --atomic so a failed upgrade rolls back automatically, and run helm diff in CI to see what will change before it happens.

Restore access first, usually by making the app use the new secret (or briefly the old one, if it is still valid), then fix the rotation process.

Typical causes: the application read the secret at startup and still uses the old value, the new value was not delivered to every consumer (some pods, a CI job, an old server), or the old secret was revoked before everything switched.

Immediate steps: confirm the error is authentication in the logs, restart or reload the application so it reads the new secret, and check every consumer of that credential.

Prevention:

• Overlap period: keep both old and new credentials valid while consumers switch (many systems support two active keys or users).
• Let applications reload secrets without a restart, or roll restarts automatically after a change.
• Know all consumers of each secret.
• Test rotation in a lower environment, and alert on authentication failure spikes.

The trap is rotating by "change it now, fix what breaks".

Test the handshake yourself with openssl s_client and read what the server presents and why the client rejects it.

Common causes:

• Expired certificate, or one that is not yet valid because of clock problems.
• Missing intermediate certificate: the server must send the full chain, or some clients fail even though a browser works.
• Hostname mismatch: the name is not in the certificate's subject alternative names.
• Missing SNI: the client does not send the server name, so the server presents the wrong certificate.
• Protocol or cipher mismatch: an old client against a server that requires TLS 1.2 or higher.
• Untrusted CA: a private CA that the client does not trust.
• Mutual TLS: the client certificate is missing or invalid.

Always test from the failing client's point of view, because browsers hide problems (like a missing intermediate) that libraries and tools do not.

Measure latency and throughput between the two hosts, and then look for packet loss and retransmissions. Do not blame the network without numbers.

Things to check:

• Distance: are they in different zones or regions? Cross-region calls add tens of milliseconds, and chatty protocols multiply that.
• Retransmissions: many retransmits point to packet loss.
• DNS time and new connection setup (TCP and TLS handshakes) that happen on every request instead of being reused.
• Proxies or service mesh sidecars adding hops.
• Bandwidth limits of the instance type.

Often the network is fine, and the real issue is making too many sequential calls, so check the number of round trips as well as the time per trip.

No. Loss at a single intermediate hop that does not continue to the final destination is usually just that router limiting its replies to probes, and it does not affect real traffic.

How to read it:

• If loss appears at hop 5 but hops 6, 7 and the destination show 0% loss, hop 5 is rate limiting ICMP replies. Ignore it.
• If loss starts at a hop and continues through all later hops to the destination, real packet loss is present from that point on.
• High latency at one hop that disappears at the next hop is also usually ICMP deprioritisation.

Run more probes (100 or more) for a stable view, and test in both directions, since paths can differ. Confirm with real traffic measures such as TCP retransmits (ss -ti, netstat -s), and check interface drops with ethtool -S and ip -s link on your own hosts.

Find out exactly where the alerting chain broke, then add alerts based on user impact rather than on internal causes only.

Possible failures:

• No rule covered this failure. Many alerts watch causes (CPU high) rather than symptoms (error rate, latency).
• The threshold or time window was too lenient.
• Monitoring was itself broken: a scrape target down means missing data, which many alert rules treat as "no problem".
• Routing or silences: the alert fired but went to the wrong channel, or was silenced or deduplicated.
• Notification path failed: the paging service or webhook was down.

Improvements:

• Alert on symptoms users feel (availability, error rate, latency), using SLO-based alerts.
• Alert on absent data and use a "dead man's switch" alert that must always fire, to prove the pipeline works.
• Test alerts regularly, including the whole paging path.

Add a fix for detection to the postmortem, because "we learned about it from customers" is a finding in its own right.

Treat the noisy alert as a bug. A page should mean "a human must act now". If it does not, change or remove it.

Ask for each noisy alert:

• Is it actionable? If the fix is "wait" or "restart", automate the fix or turn it into a ticket.
• Is it urgent? If it can wait until morning, send it to a ticket or a chat channel, not a pager.
• Does it measure user impact? Prefer symptom-based alerts over raw resource thresholds.
• Can the rule be tuned? Add a duration ("for 10 minutes"), group related alerts, raise the threshold to a meaningful level, and add inhibition so one root cause does not send 50 pages.

Track the alerts per shift and the share that needed action. Constant false alarms cause alert fatigue, and then real incidents get ignored.

Also fix the cause of the repeated issue: a recurring alert often points to a known problem that nobody has prioritised.

Follow the log path hop by hop and find where the data stops: application, collector agent, transport, storage and search.

1. Does the app write logs? Check stdout or the log file directly. In containers, apps should log to stdout.
2. Is the agent running (Fluent Bit, Fluentd, Filebeat, the OpenTelemetry collector)? Check its logs for errors, restarts and buffer warnings.
3. Is the agent reading the right files, and does it have permission? Log rotation can also break file tailing.
4. Backpressure: a full buffer, a slow or down destination, or dropped records.
5. Destination limits: rate limits, index or storage quota, rejected documents because of a mapping error.
6. Timestamps: logs may exist but with a wrong timestamp, so you search the wrong time window.
7. Retention or an index lifecycle policy deleted them.

The trap is searching only in the log UI. Check the agent's own health and metrics first.

Check its state, what system call it is blocked in, and what it is waiting on (disk, network, a lock or another process).

Interpretation:

• Stuck in futex or pthread_cond_wait: waiting on a lock, possibly a deadlock.
• Stuck in read or recvfrom on a socket: waiting for a remote service that never answered, so look at that connection in lsof and ss.
• State D: blocked in I/O, probably a hung disk or network filesystem.

For managed runtimes, take thread dumps (jstack, kill -3 for Java, py-spy dump for Python), and take several a few seconds apart to see what is not moving.

Check when it rebooted and what the logs say just before the restart, then separate software, hardware and cloud causes.

Possible causes:

• Kernel panic or a crash: look for panic messages, and check if kdump produced a crash dump.
• Out of memory with panic_on_oom, or a hung system that a watchdog rebooted.
• Hardware problems: memory or disk errors, power loss (check hardware logs like IPMI).
• Cloud events: host maintenance or retirement. Check the provider's event log and notifications.
• Automatic updates that reboot the machine (for example unattended upgrades).
• A human or automation: check shell history, audit logs and CI jobs.

If the previous boot's log is missing, enable persistent journald storage so the next reboot leaves evidence.

First decide which memory is growing: the Java heap, or non-heap memory (metaspace, threads, native buffers). Then capture evidence while the problem is present.

Steps:

1. Watch the heap with monitoring or jstat -gcutil <pid> 1000. A heap that stays high after full garbage collections suggests a leak.
2. Capture a heap dump: jmap -dump:live,format=b,file=heap.hprof <pid>, or start the JVM with -XX:+HeapDumpOnOutOfMemoryError so you get one automatically at the crash.
3. Analyse it with Eclipse MAT or a similar tool: look for the "dominator tree" and "leak suspects", for example a growing cache, a static list or unclosed resources.
4. Compare the container limit with the JVM settings. The container limit must be larger than -Xmx plus metaspace, thread stacks and native memory. Otherwise the container is OOM-killed without a Java OutOfMemoryError.

Use -XX:MaxRAMPercentage for container-aware sizing, and enable Native Memory Tracking if the heap looks normal but the process keeps growing.

Revoke and rotate the key immediately, then investigate whether it was used. Removing it from the repository is not enough, because bots scan public repositories within minutes.

Steps, in order:

1. Disable or delete the key (and create a new one for the legitimate use). Do this first.
2. Check usage: review CloudTrail or the provider's audit logs for activity with that key during the exposure window. Look for new users, instances (often crypto-mining), or data access.
3. Contain any damage: terminate unknown resources, remove unauthorised access, and rotate other credentials the key could reach.
4. Clean up the repository (rewrite history if needed), knowing that forks and clones may still hold the secret. Treat it as compromised regardless.
5. Report to the security team as an incident.

Prevention: pre-commit secret scanning, repository secret scanning and push protection, short-lived credentials (roles and OIDC federation instead of long-lived keys), and least-privilege permissions to reduce the blast radius.

Confirm the outage, limit its effect on your users, and communicate. You cannot fix the vendor, but you can control how much their failure hurts you.

Immediate steps:

1. Confirm using the vendor's status page and your own error metrics.
2. Degrade gracefully: use cached data, a fallback provider, a queue to retry later, or disable the feature that needs it and show a clear message.
3. Protect yourself: make sure timeouts and circuit breakers are working so slow calls do not exhaust your threads or connections and cause a cascading failure.
4. Communicate to your users and your support team, and open a ticket with the vendor.
5. Recover safely: when the vendor comes back, release the queued work gradually to avoid overload.

Design lessons: every external call needs a timeout, retry limits and a fallback plan, and critical dependencies deserve a second provider or an acceptable degraded mode. Check your own SLA, since your availability cannot exceed that of a hard dependency.

Prefer actions that change the system over actions that rely on people remembering. Rank ideas by how strongly they prevent, detect or limit the problem.

A useful order, strongest first:

1. Remove the hazard or make the mistake impossible (guardrails, safer defaults, removing the risky manual step).
2. Automate the check or the fix (validation in CI, automatic rollback, self-healing).
3. Detect faster (better alerts on symptoms, dashboards, canaries).
4. Limit the blast radius (gradual rollouts, isolation, rate limits, quotas).
5. Improve response (runbooks, practice drills, clearer ownership).
6. Training or reminders, which are weakest because people forget and change teams.

For every action ask: "if this exact situation happened again, would this have stopped or shortened it?" Give each action an owner, a deadline and a place in the normal backlog. Actions without owners are a common reason incidents repeat.

Collect enough data continuously so that the next occurrence explains itself, and look for patterns in when it happens.

Look for correlations in the timing:

• Scheduled jobs (cron, backups, batch processing, log rotation).
• Traffic patterns by hour or day.
• Garbage collection, memory growth or certificate and token expiry cycles.
• Deployments, autoscaling events or node replacements.
• A specific customer, request type or dependency.

Capture evidence automatically:

• Add targeted metrics, logs or traces around the suspicious area.
• Run an automatic snapshot script that triggers on a symptom (for example, when latency crosses a threshold it collects top, ss -s, a thread dump and iostat output).
• Keep higher-resolution metrics, since a 5-minute average can hide a 20-second spike.

Compare a bad period with a good one, side by side. The wrong approach is waiting for the problem to happen again and starting to look then, because it will already be gone.

Linux load average counts processes that are running and processes stuck in uninterruptible sleep (state D, usually waiting for disk or network storage). So a high load with idle CPUs usually means processes are blocked on I/O.

Typical causes: a slow or throttled disk (for example a cloud volume that hit its IOPS limit), a hung NFS or network mount, a failing disk, or a storage backend problem.

The common wrong answer is to say "the CPU is idle so the server is fine". Load average is about demand for the system, not only CPU. Fix the storage bottleneck, not the CPU.

Brendan Gregg: Linux Load Averages

The most likely cause is deleted files that a running process still holds open. The filesystem cannot release the space until the last file handle is closed, so df still counts it while du cannot see the file.

This often happens when someone runs rm on a big log while the application is still writing to it, or when log rotation does not signal the process to reopen its log file.

Fixes:

• Restart or reload the process that holds the file, which releases the space.
• Without a restart, truncate through the file descriptor: : > /proc/<pid>/fd/<n>.

Other reasons for a mismatch: data hidden under a mount point (files written to a directory before a volume was mounted over it), and reserved blocks on ext4. Fix the rotation setup so it does not happen again.

First find where in the request path the time is spent, then look at what changed. Do not guess a cause before you know the slow layer.

1. Scope it. Is every endpoint slow or only one? Every region or one? All users or some? A narrow scope narrows the cause.
2. Check the golden signals: latency, traffic, errors, saturation. Did traffic grow? Did errors rise too?
3. Use traces to see which span is slow: the app itself, the database, a cache, or an external API.
4. Check resource saturation on each tier: CPU, memory, disk I/O, network, connection pools, thread pools.
5. Look at changes: deploys, config, feature flags, data growth, a new customer, a cloud incident.
6. Check dependencies: slow queries, cache misses (hit rate dropped), a degraded third-party service.

The trap is looking only at the application server's CPU. Many latency problems live in a dependency or in queueing, not in the app's own computation.

Brendan Gregg: The USE Method

Averages hide tail latency. If 1% of requests take 5 seconds, the average barely moves, but with many requests per page or per session, almost every user hits a slow one.

Look at p95, p99 and p99.9 using histograms, and break them down by instance, endpoint and time.

Common causes of slow tails:

• Garbage collection pauses.
• One bad or overloaded instance among many.
• Cold caches or cold starts.
• Lock contention and queueing under bursts.
• Retries and timeouts adding seconds.
• Slow DNS lookups or new TLS connections.
• A noisy neighbour on shared hardware.

Method: find slow requests in traces, then compare them with fast ones (same endpoint, which host, which dependency). If a request fans out to many backends, the slowest backend decides the total, so tails get amplified.

Never report only the average in an incident.

Split the failing requests by every dimension you can until the pattern appears. Intermittent errors almost always have a hidden pattern.

Slice the access logs and metrics by:

• Instance or pod: one bad node would give a steady slice of errors proportional to its share of traffic.
• Availability zone or region.
• Endpoint, method and payload size.
• Client, user agent or customer.
• Time: does it line up with a cron job, GC, or a deploy?
• Upstream address: log the backend that handled each request.

Other classic causes: a deployment with mixed versions, one backend with a stale DNS or config, connection reuse races (a keepalive connection closed by one side just as the other sends), and retries from a downstream dependency.

If errors are too rare to catch, add more logging, or capture traffic with tcpdump on the suspect host. The wrong approach is to look only at overall averages, since 2% disappears into the noise.

Rolling back the code may be unsafe if the old version cannot work with the new schema or data. Decide based on what the migration did and whether the old code still functions.

Options:

• Roll back the code only if the migration was backward compatible (for example, added a nullable column). This is the safe case.
• Roll forward with a fix, when the change cannot be undone.
• Disable the new behaviour with a feature flag while keeping the schema.
• Restore data from a backup or point-in-time recovery only as a last resort. It loses writes made since the migration.

The real answer is prevention: use the expand and contract pattern. First add new columns or tables while old code still works (expand), deploy code that uses both, migrate the data, and only later remove the old structure (contract). Each step is then reversible.

The trap is a destructive migration (drop or rename column) in the same release as the code change.

Test each resolver separately and look for packet loss, because intermittent DNS failures usually come from one bad resolver in the list or from dropped UDP packets.

Run the loop against each resolver in /etc/resolv.conf and compare. Also check:

• Rate limits: cloud DNS resolvers limit packets per second per network interface. A burst of lookups can be throttled.
• Conntrack race on Linux: parallel A and AAAA queries from the same socket can be dropped and retried after 5 seconds. This shows as a 5-second delay.
• Search domains and ndots: each name can create several lookups, multiplying load (common in Kubernetes).
• Network issues: capture with tcpdump -i any port 53 and look for queries without answers.

Mitigations: a local caching resolver on each node, fewer lookups, more reliable resolvers, and TCP fallback for large responses.

You are likely running out of ephemeral ports. Every outbound TCP connection to the same destination IP and port needs a unique local port, and a closed connection keeps its port reserved in TIME_WAIT for about 60 seconds.

If each request opens a new connection, around 28,000 connections per minute to one destination exhausts the range.

Fixes, best first:

1. Reuse connections: HTTP keep-alive and connection pools.
2. Widen ip_local_port_range.
3. Spread traffic over more destination IPs or source IPs.
4. In clouds, check the NAT gateway: it has its own port limit per destination.

The wrong fix is tuning dangerous tcp_tw_* settings first. Connection reuse in the application is the real solution.

NotReady means the control plane stopped getting healthy heartbeats from the node's kubelet. Find out whether the node, the kubelet, the network or the runtime failed.

Common causes: the kubelet crashed, the container runtime is down, the disk or memory is exhausted, the CNI network plugin failed, or the VM itself died or lost network.

What happens to pods: after the node stays unreachable (about 5 minutes by default through taint-based eviction), the control plane marks its pods for eviction and controllers recreate them on healthy nodes. Pods with local data or single replicas suffer downtime.

Mitigation: cordon the node, drain it if it responds, and replace it rather than repairing it, especially in autoscaled clusters.

Common causes are the ndots:5 setting multiplying lookups, an overloaded CoreDNS, and conntrack races on UDP.

Pods get ndots:5 in /etc/resolv.conf. Any name with fewer than five dots (like api.example.com) is first tried with every search domain (api.example.com.default.svc.cluster.local, and so on) before the real name, so one lookup becomes four or five, with failures in between.

Check:

Fixes:

• Use fully qualified names with a trailing dot (api.example.com.) or lower ndots in the pod's dnsConfig.
• Scale CoreDNS and raise its resource limits.
• Run NodeLocal DNSCache so lookups are served on the node and use TCP upstream.
• Cache lookups in the application.

Many apps look slow only because of DNS, so measure the lookup time separately.

Find the offending query and what is blocking it, stop the damage, then fix the cause. The fix for the outage and the fix for the query are separate steps.

Find it:

MySQL has the slow query log and SHOW PROCESSLIST.

Immediate response: cancel or kill the runaway query (pg_cancel_backend), or kill the blocker holding a lock. Shift reads to a replica if possible.

Typical root causes: a missing index after the table grew, a query plan that changed after new statistics, a lock held by a long transaction, or an unbounded query from a new feature. Check with EXPLAIN (ANALYZE). Add the index with CREATE INDEX CONCURRENTLY so you do not block writes.

A cache stampede happens when a popular cache entry expires and many requests all miss at once and hit the database together. The database gets overwhelmed, slows down, and the cache stays empty longer, which makes it worse.

Prevention techniques:

• Request coalescing (single flight): only one request rebuilds the value, and the others wait for it.
• Locking or a short "lease" on the key while it is being refreshed.
• TTL jitter: add random time to each expiry so keys do not expire together.
• Stale-while-revalidate: serve the slightly old value while one worker refreshes in the background.
• Pre-warming important keys before expiry or after a restart.

A related problem is a cold cache after a deploy or cache restart, where everything misses at the same time. Put a rate limit in front of the database so it can survive a cold start.

The wrong answer is simply setting a longer TTL. That only delays the stampede.

A retry storm is when failing requests are retried by many clients, multiplying the load on an already struggling service so it cannot recover.

During the incident: reduce incoming load (rate limit, shed traffic, disable the retrying feature), and let the service breathe before restoring traffic gradually.

Prevention in code and design:

• Exponential backoff with jitter, so clients spread out instead of retrying together.
• Retry budgets: limit retries to a small fraction of normal traffic.
• Circuit breakers that stop calling a failing dependency.
• Retry at one layer only. Three layers retrying three times each means 27 calls per request.
• Only retry idempotent operations.

A common wrong answer is "add more retries to improve reliability". Retries help with brief glitches but cause outages when the failure lasts.

A cascading failure is when a problem in one component spreads to others, usually because callers wait on a slow dependency, run out of threads or connections, and then fail themselves.

How it spreads: service A waits for slow service B, A's worker threads fill up waiting, A stops answering its own callers, and the failure moves up the chain.

Stop it during the incident:

• Cut or reduce traffic to the sick service (rate limits, load shedding).
• Turn off non-essential features that call it.
• Restart or scale carefully, because a restarted service facing full traffic can fall over again.

Design defences:

• Timeouts on every remote call.
• Circuit breakers to fail fast.
• Bulkheads: separate thread or connection pools per dependency.
• Load shedding and graceful degradation (return cached or partial data).
• Capacity headroom, so one failure does not push everyone else over the limit.

The trap is fixing only the visible failing service while the real cause is the missing timeouts upstream.

Scaling is reactive, so by the time CPU is high and new instances finish starting, the spike has already hurt users. Reduce the total reaction time and scale on earlier signals.

The delay has several parts: metric collection delay, the rule's evaluation period, instance or pod startup time, image pull, and application warm-up.

Improvements:

• Use leading indicators like request rate, queue depth or concurrent connections, instead of CPU.
• Lower the target utilisation (for example 50% instead of 80%) to keep headroom.
• Scale up fast, scale down slowly.
• Predictive or scheduled scaling for known peaks (a daily rush, a campaign).
• Cut startup time: smaller images, pre-pulled images, warm pools of ready instances.
• Protect the service meanwhile with rate limiting, queues and graceful degradation.

The trap is to only tune thresholds. If startup takes five minutes, no threshold will save you from a spike that grows in thirty seconds.

Confirm the scope, move traffic away from the affected zone or region, and rely on capacity that already exists. Do not depend on actions that need the broken part to work.

Steps:

1. Confirm: provider status page and your own metrics. Is it one zone, a service, or a whole region?
2. Shift traffic: drain the affected zone from load balancers, change DNS or traffic weights to another region, and use health-based failover.
3. Use existing capacity. During a large outage everyone tries to launch instances at once, and control planes may be slow or failing. A design where the other zones already run enough capacity to carry the load is called static stability.
4. Protect data: check replication state before promoting a standby, and know your RPO.
5. Communicate: update status pages and stakeholders.
6. Fail back carefully after recovery.

The trap is an untested failover plan. Practise it with game days, and keep runbooks that do not depend on the failing region (for example the CI system or the secrets store in the same region).

Stop and do not apply. A destroy and create on a stateful resource means data loss unless you prove otherwise, so find out exactly why Terraform wants to do it.

Read the plan: it marks the attribute that "forces replacement". Typical reasons:

• You changed an attribute that cannot be updated in place (some engine, name or subnet settings).
• You renamed or moved the resource in code, so Terraform thinks the old one is gone and a new one is needed.
• A provider upgrade changed how a value is read.
• The state does not match the real resource.

Fixes: for renames or refactors use a moved block or terraform state mv. Revert the code change if the attribute should not change. Take a backup or snapshot before any risky change.

Safeguards: set lifecycle { prevent_destroy = true } on critical resources, turn on deletion protection in the cloud service, and require human review of plans that contain destroys.

Suspect an MTU or path MTU discovery problem. Small packets pass, but larger packets are dropped somewhere on the path because they exceed a link's maximum size and the "fragmentation needed" message never reaches the sender.

This often appears with VPNs, tunnels, overlay networks (VXLAN, IPsec, GRE) and some cloud links, which add headers and reduce the usable MTU.

Test:

If 1472 fails but a smaller size such as 1400 works, you have found it.

Typical symptoms: the TCP connection opens fine (small packets), TLS handshakes stall (the certificate message is big), and then large responses hang.

Fixes: allow ICMP "fragmentation needed" through firewalls, lower the MTU on the interface or tunnel, or use TCP MSS clamping on the gateway. Blocking all ICMP is the classic cause of this problem.

High cardinality means a metric has a huge number of unique label combinations, and every combination is a separate time series that Prometheus must hold in memory. Memory use grows with the number of active series.

Typical cause: a label with unbounded values, such as user IDs, request IDs, full URLs with parameters, or pod names that change constantly.

Find the culprits:

The TSDB status page in the Prometheus UI also lists top metrics and labels.

Fixes:

• Remove or replace the offending label (use route templates like /users/{id} instead of raw paths).
• Drop series or labels at scrape time with metric_relabel_configs.
• Set sample_limit or label_limit on scrape jobs so one bad target cannot hurt everyone.
• Move per-user detail to logs or traces, which handle high cardinality better.

Adding RAM only delays the problem, so find the label first.

Enable GC logging and compare pause times with the latency spikes. If the spikes line up with GC pauses, the collector is the cause.

For Java:

Look for long "stop the world" pauses, frequent full GCs, a heap that stays almost full after collection, and a very high allocation rate.

Typical causes and fixes:

• Heap too small: the collector runs constantly. Increase the heap, within the container limit.
• Too much allocation: reduce garbage in hot paths (object churn, big temporary buffers).
• Wrong collector: consider G1 with a pause target, or ZGC or Shenandoah for low-latency needs.
• Memory leak: the heap fills up with live objects, so collection cannot free memory.

The same idea applies to Go, .NET and others: check the runtime's GC metrics against latency graphs.

The trap is blaming the network or the database before checking the process's own pauses. Also note that a container CPU limit can throttle GC threads, so check CPU throttling metrics too.

When all worker threads are blocked, the process can still accept connections but never finish any work. Thread dumps show what each thread waits for.

Take three thread dumps a few seconds apart (jstack <pid> or kill -3 for Java, py-spy dump for Python, SIGQUIT for Go goroutines). Then look at the pattern:

• Many threads in the same stack, waiting on a socket read: a downstream service is slow and there is no timeout, so threads pile up.
• Threads waiting for a connection from a pool: the pool is exhausted, perhaps by leaked connections.
• Threads blocked on locks, with a cycle (A waits for B, B waits for A): a deadlock. Java's jstack reports detected deadlocks directly.

Mitigation is a restart (take the dumps first). Fixes: set timeouts on every outbound call, use separate pools (bulkheads) per dependency, bound queue sizes so work is rejected instead of piling up, and keep lock ordering consistent.

The trap is restarting before capturing a dump, which destroys the only evidence.

Stop further damage, preserve what you can, and restore from the best available source, while communicating clearly. Avoid panic commands that could make recovery harder.

1. Stop the bleeding: pause the automation or the person who might repeat the action, and stop writes that could overwrite recoverable data.
2. Assess: what was deleted, when, and what recovery options exist.
3. Restore options, from best to worst: the resource's built-in protection (soft delete, recycle bin, deletion protection snapshots), point-in-time recovery, the latest backup or snapshot, and replicas that did not replay the delete.
4. Declare an incident and communicate the expected data loss window (your real RPO).
5. Verify the restored data before sending traffic to it.

Afterwards, run a blameless postmortem. The system allowed one person to delete production, so add guards: deletion protection, separate roles for destructive actions, approval for risky operations, and backups that are tested and stored separately from the production account.

An untested backup is only a hope. Restores fail because backups were never verified: they were corrupted, incomplete, encrypted with a lost key, or missing pieces such as transaction logs.

Typical causes:

• The backup job reported success but produced an empty or partial file.
• Missing log files (WAL or binlogs) needed for point-in-time recovery.
• The encryption key is unavailable or in the destroyed environment.
• The restore procedure was never documented or practised, and takes much longer than expected.
• Backups lived in the same account or region as the thing that failed.
• Version or configuration differences stop the restore.

During the incident, try an earlier backup, and use any replica or snapshot that still exists.

Prevention:

• Restore regularly, automatically, into a test environment and run integrity checks.
• Measure real restore time against your RTO, and data loss against your RPO.
• Keep copies in another account or region, and keep keys separate and recoverable.
• Alert when a backup is missing or suspiciously small.

Isolate it, preserve evidence, and involve your security team. Do not immediately wipe or reboot it, because that destroys the evidence you need to learn what happened.

1. Contain: remove it from the load balancer and restrict its network access (change its security group to block traffic, but keep the machine running if you can). Avoid alerting an attacker unnecessarily, and avoid actions that destroy memory state.
2. Preserve evidence: take a disk snapshot, and capture memory and logs if you have the tooling. Record the time and actions you take.
3. Investigate (ideally on a copy): unknown users or SSH keys, cron jobs and systemd units (persistence), unusual processes and outbound connections, modified binaries, and authentication logs.
4. Assume credentials are stolen: rotate secrets, keys and tokens that were on the server or reachable from it.
5. Rebuild from a known-good image instead of cleaning the old system, and close the entry point (patch, fix the exposed service or leaked credential).
6. Report and write a postmortem, and follow legal and compliance duties.

The trap is "clean it and move on". You will not be sure the attacker is gone.

Look at who is sending the traffic and what they ask for. The traffic pattern tells you whether it is a marketing success, a bot, or an attack.

Analyse the logs and edge metrics by:

• Source IP, ASN and country (a huge number of unique IPs, or a sudden new geography).
• User agent and request headers (identical or missing).
• Paths (everyone hitting one expensive endpoint like search or login).
• Rate per client and the share of errors or cache misses.
• Timing: does it start sharply, or follow a known event such as a campaign, a news mention or a sale?

Responses:

• Legitimate spike: scale out, serve from cache and CDN, and shed non-essential features.
• Attack: use a CDN and WAF, add rate limits, block or challenge abusive sources, enable the DDoS protection service of your provider, and protect expensive endpoints.

Also confirm that your autoscaling and database can survive it, because unbounded autoscaling during an attack can produce a huge bill. Decide in advance who may enable emergency protections.

Find out what is different about production, and use the data production gives you. Common differences explain most "only in prod" bugs.

Typical differences:

• Data: volume, unusual values, old records, or encoding.
• Scale and concurrency: race conditions and timeouts that appear only under real load.
• Configuration and secrets: different settings, feature flags or permissions.
• Infrastructure: network topology, DNS, TLS, proxies, load balancers, instance sizes, and versions of the OS or dependencies.
• Traffic: real clients and unusual requests that tests never send.

How to investigate:

1. Gather evidence from production: logs, traces, request IDs and metrics for failing requests.
2. Compare environments systematically (configuration, versions, data shape).
3. Try to reproduce with production-like data and load, or replay captured traffic against a staging copy.
4. Add safe, targeted extra logging or use a feature flag to enable debugging for a small group.
5. Use a canary or shadow traffic to test a fix with limited risk.

Avoid making changes directly on production without a plan, and avoid exposing sensitive data in debug output.