How to Troubleshoot a Linux Production Server

At 2am an alert fires. Your payment API is timing out. Users are seeing 503 errors. You SSH into the server and open top.

Most engineers stare at it, pick the process with the highest CPU, and guess. That approach takes 20 minutes. This one takes 4 — because it follows a fixed decision tree instead of improvising under pressure.

The 7-Step Diagnostic Sequence

Work through these steps in order. Every check narrows the problem. Stop at the step where you find the culprit — you rarely need to reach step 7.

Step 1 — Get the System Snapshot

Run these four commands immediately. They give you the full picture in 30 seconds.

uptime shows the load average — three numbers representing 1, 5, and 15-minute windows.

A load average above your CPU core count means the system is overloaded. On a 4-core server, 8.42 means twice as many processes are waiting to run as the CPU can service. Run nproc to confirm your core count.

free -h reveals memory pressure. Watch available, not free — when available drops below 10% of total RAM, the kernel begins swapping aggressively to disk. That is catastrophic for databases and APIs.

df -h catches a full disk before you waste 10 minutes elsewhere. A full disk causes processes to crash silently with no obvious log trail.

Step 2 — Diagnose CPU Saturation

If load average is above core count, find what is consuming it.

The %CPU column shows the culprit immediately. At Swiggy, a misconfigured background job consuming 97% CPU caused the order API to queue requests for 8 seconds during a dinner spike — ps aux found it in 15 seconds.

Reading the CPU output pattern:

The iowait percentage from top is the most commonly missed signal. If %wa is above 20, the CPU is waiting on disk. Tuning the application will not help — you need to address disk I/O first.

For historical trend context beyond the current snapshot, sar -u 1 5 (from the sysstat package) shows CPU behavior over the last few minutes rather than a single instant — useful for confirming whether a spike is sustained or already passing.

Step 3 — Diagnose Memory and OOM Events

When the kernel runs out of memory it invokes the OOM killer — it terminates the process with the highest memory score to reclaim RAM. The victim is almost never the process you would expect.

A real OOM entry in dmesg looks like this:

The score 821 indicates how aggressively the kernel targeted that process — higher scores get killed first. When you see this at Zerodha for example, it is almost always the analytics aggregation job — not the trading engine — consuming memory unnoticed until it starves everything else.

To find which processes are currently consuming swap:

Step 4 — Diagnose Disk Issues

A full disk is the sneakiest failure mode. Processes write nothing, log nothing, and silently return errors that look like application bugs.

The inode trap is frequently missed. You can have 40GB of free disk space and zero inodes remaining. Every open() system call will fail with ENOSPC — the same error as a full disk. The difference only appears in df -ih.

To locate the directory generating the most files:

For iostat, watch the %util column — above 80% means the disk is saturated. The await column shows average I/O wait time in milliseconds. Above 20ms for SSDs or 100ms for spinning disks indicates queueing that will slow every process touching that device.

Step 5 — Diagnose Network and Connections

Networking failures masquerade as application failures. What looks like a hanging API is often a connection that cannot be established.

Connection state guide:

Test connectivity to a specific dependency directly from the server:

This isolates whether the problem is application-level or infrastructure-level in one command. If the TCP connection fails, the application cannot fix it — you have a network, firewall, or service issue.

Step 6 — Read the Logs

By step 6 you know which component is under pressure. Now find the specific error that triggered the incident.

The most important question is: when did the first error appear?

The timestamp of the first error tells you where to look for the cause. What changed in the 2-5 minutes before that line?

Step 7 — Find What Changed

Eight out of ten production incidents trace back to a change in the last 2 hours. Always end your investigation by confirming what changed.

If the server runs a cron job, check whether it fired recently:

A Real Incident: Razorpay Checkout Latency

Symptom: Checkout API p99 latency spiked from 180ms to 4200ms at 01:47. Payment failure rate rose to 12%.

Root cause: PostgreSQL autovacuum acquired a lock on the 48GB payments table during peak checkout traffic. Every query to that table queued behind the lock for 3 minutes 40 seconds.

Total investigation time: 4 minutes. The fix — cancelling the autovacuum and scheduling it for off-peak hours — took 2 minutes.

The Full Diagnostic One-Liner Sequence

Copy this block when an alert fires. Run it top to bottom.

Production Implementation Guidelines

Run this sequence on every P1 alert before escalating. Systematic diagnosis reduces MTTR from 20-30 minutes to under 5 minutes — the difference between one missed SLO and ten.

For teams running on Indian cloud infrastructure, iostat is especially relevant on gp2 EBS volumes in ap-south-1 — the Mumbai region's gp2 IOPS burst bucket depletes quickly under sustained write loads, causing iowait spikes that are invisible to application-level monitoring.

Configure a motd or alias for this sequence on every production bastion host so any engineer can run it without looking it up during an incident. Print or pin the decision tree from Step 1-7 somewhere visible — the value of a systematic sequence is that a junior engineer can follow it correctly at 2am without having memorized it first.

Note

References and Further Reading

• Linux Performance Analysis in 60 Seconds — Netflix Tech Blog
• USE Method by Brendan Gregg — Utilisation, Saturation, Errors framework
• systemd journalctl Manual — Official reference