Linux Internals and Performance Interview Questions

A process has its own address space and resources, while threads inside one process share the same address space, open files and heap, and each thread has its own stack and registers.

Under the hood Linux treats both as "tasks". A new process and a new thread are both created with the clone() system call. The only difference is which resources the new task shares with its parent. A thread shares memory, file descriptors and signal handlers. A process created with fork() gets a copy of them.

This has practical effects:

• A bug in one thread (a bad pointer, for example) can crash the whole process, because they share memory. Two separate processes are isolated from each other.
• Each thread gets its own thread ID (TID) and is scheduled independently by the kernel. ps -eLf lists threads, and the NLWP column is the thread count.
• Threads are cheaper to create and to switch between, because there is no new address space to set up.

The common wrong answer is "threads are not scheduled by the kernel". In Linux they are. The kernel sees every thread as a separate schedulable task.

fork() creates a new process that is a copy of the caller, and exec() replaces the program running inside an existing process with a different program.

A shell starts a command by combining them. First fork() makes a child that is a near-identical copy of the shell. Then the child calls execve("/bin/ls", ...), which throws away the copied shell code and loads ls into the same process. The PID does not change on exec, because no new process is created. The program running in it is swapped.

They are separate steps for a good reason. Between fork and exec the child can change its own setup before the new program starts: redirect output to a file, close descriptors, change the working directory or drop privileges. That is how shell redirection and pipes work.

fork() is cheap because of copy-on-write. The child shares the parent's memory pages until one of them writes to a page, and only then is that page copied.

The letter in the STAT column of ps (or the S column in top) is the process state:

Most processes on a healthy server are in S. A few in R is normal. The two states worth investigating are D and Z. Many processes stuck in D point to slow or broken storage, and many Z entries point to a parent that is not cleaning up its children.

A quick way to list the suspicious ones:

A zombie is a process that has already exited but still has an entry in the process table because its parent has not called wait() to collect its exit status.

It uses almost no memory. All that remains is the PID and a small record holding the exit code. You cannot kill it with kill -9, because it is already dead. Signals do nothing.

The fix is to make the parent reap it:

• Send SIGCHLD to the parent (kill -CHLD <parent-pid>). A well-written parent will reap on that signal.
• If the parent is buggy, restart or kill the parent. The zombie is then adopted by PID 1 (init or systemd), which reaps it immediately.

A handful of short-lived zombies is normal. The real danger is a parent that leaks them without limit, because every zombie holds a PID and the system has a finite PID space (kernel.pid_max). Thousands of zombies usually mean a bug in the parent program, so fix that program rather than cleaning up by hand each time.

Virtual memory is a layer between programs and physical RAM where every process gets its own private range of addresses, and the CPU's memory management unit translates those virtual addresses to physical memory using page tables the kernel maintains.

Memory is split into fixed-size pages, normally 4 KiB. The kernel maps each virtual page to a physical page, or to nothing yet.

Why it exists:

• Isolation. One process cannot read or write another's memory, because its addresses simply do not map there.
• Simplicity. Every program can be built as if it owns a large, continuous address space.
• Efficiency. Pages are only given real RAM when first touched. Shared libraries are loaded once and mapped into many processes. Files can be mapped into memory with mmap.
• Overcommit. Programs can reserve more address space than there is RAM, and the kernel provides real pages on demand.

An address that has no valid mapping causes a page fault, and an invalid access ends in a segmentation fault. This is the foundation for understanding RSS, swap, page cache and the OOM killer.

A file descriptor is a small integer that a process uses to refer to an open file, socket, pipe or device, and it is an index into that process's table of open files.

Linux follows the idea that "everything is a file", so one set of calls (read, write, close) works on regular files, network sockets, pipes and terminals. When a process opens something, the kernel returns the next free number, and the process uses that number from then on.

Every process starts with three:

Redirection works by changing what these numbers point to. command > out.txt makes descriptor 1 point at the file, and 2>&1 makes descriptor 2 point wherever descriptor 1 points.

You can see a process's open descriptors under /proc:

Descriptors are a limited resource. A program that opens connections or files and forgets to close them (a descriptor leak) eventually hits "Too many open files".

A system call is how a program asks the kernel to do something it is not allowed to do itself, such as reading a file, opening a socket or creating a process, and it is the only door between user space and kernel space.

The CPU runs code in two privilege levels:

• User space. Ordinary programs run here. They cannot touch hardware or other processes' memory directly.
• Kernel space. The kernel runs here with full access to hardware and all memory.

When a program calls read(), the library places the arguments in registers and executes a special instruction. The CPU switches to kernel mode, the kernel does the work, then returns to user mode with the result. This switch costs some time, which is why programs try to avoid making thousands of tiny system calls (for example by buffering writes).

Common calls include open, read, write, close, mmap, fork, execve, socket, accept and epoll_wait. A few cheap ones, such as reading the clock, are handled by the vDSO, a small kernel-provided library that avoids the full switch.

Knowing this helps in operations work, because tools such as strace show exactly which system calls a program makes, and that tells you what it is doing at the boundary with the OS.

/proc is a virtual filesystem the kernel uses to expose live information about processes and the system, so reading a file there asks the kernel for current data and nothing is stored on disk.

Each running process has a directory /proc/<pid>/. System-wide data sits at the top level.

Many familiar tools are simply readers of /proc. ps, top, free, lsof and uptime all parse these files.

A related filesystem is /sys, which exposes devices, drivers and cgroups, for example /sys/block/sda/queue/scheduler or /sys/fs/cgroup/. Knowing both lets you inspect a system even when the usual tools are missing, such as inside a minimal container.

An inode is the data structure a filesystem uses to describe one file or directory, holding everything about it except its name and its contents.

An inode stores the owner and group, permission bits, file size, timestamps, the link count, and pointers to the data blocks on disk. The file's name lives in a directory entry that points to the inode number. That separation is why hard links work: two names can point to the same inode.

This explains several behaviours:

• Hard link vs symlink. A hard link is another name for the same inode. A symlink is a small separate file containing a path.
• Delete semantics. rm removes a name and decrements the link count. The data is freed only when the count reaches zero and no process has the file open.
• Rename is cheap. mv within one filesystem only changes the directory entry, and moves no data.

A filesystem has a fixed or dynamic number of inodes. On ext4 the number is set when the filesystem is created, so it is possible to run out of inodes while plenty of space remains, which is the next question.

SIGTERM asks a process to exit and lets it clean up, SIGKILL ends it immediately and cannot be caught, and SIGHUP traditionally means "hang up", which many daemons use as a request to reload their configuration.

kill -9 is a last resort because the process gets no chance to run any cleanup. Temporary files stay behind, locks are not released, buffered data is not flushed, child processes can be orphaned, and a database may need crash recovery on next start.

The sensible order is: send SIGTERM, wait a reasonable time, and only if the process is still alive, send SIGKILL. Container platforms follow exactly this pattern, with a grace period between the two.

Two edge cases to know. A process in D state will not die from either signal until its I/O finishes. And SIGKILL cannot be delivered to a zombie, since it is already dead.

signal(7) manual page

The machine runs firmware, which starts a bootloader, which loads the kernel, which starts the first process (PID 1, usually systemd), which brings up all the services.

1. Firmware. UEFI (or older BIOS) tests the hardware and finds a boot device.
2. Bootloader. GRUB loads the kernel image and an initial RAM filesystem (initramfs) into memory, and passes the kernel command line.
3. Kernel start. The kernel initialises CPUs, memory and drivers. The initramfs contains just enough drivers and tools to find and mount the real root filesystem, which may be on LVM, RAID, or an encrypted device.
4. PID 1. The kernel starts /sbin/init, which is normally systemd. It reads unit files and starts services in dependency order until it reaches the default target, such as multi-user.target.
5. Login. A getty or the display manager presents the login.

Useful commands when boot is slow or broken: systemd-analyze blame shows which units took the time, journalctl -b shows this boot's logs, and the GRUB menu lets you edit the kernel command line (for example to boot into rescue mode).

The shell parses the line, creates a pipe, forks one child per command, wires each child's input and output to the pipe, runs each program with exec, and then waits for them to finish.

Step by step:

1. The shell splits the line into two commands joined by a pipe.
2. It calls pipe(), which gives it two file descriptors: one end to write into, one end to read from.
3. It forks twice. In the first child it uses dup2() to point standard output (fd 1) at the pipe's write end. In the second child it points standard input (fd 0) at the read end.
4. Each child calls execve() after searching the directories in $PATH for the program.
5. The parent shell closes its copies of the pipe and calls wait4() until both children exit, then prints the next prompt.

The two programs run at the same time. grep starts reading as soon as ls writes the first bytes, and a full pipe buffer blocks the writer until the reader catches up.

PID 1 has two special jobs, reaping orphaned children and handling signals differently, and most applications do not do either job, so running one as PID 1 in a container causes zombies and containers that ignore docker stop.

First, orphans. When a process dies, its surviving children are re-parented to PID 1, which must call wait() on them. A normal init system does this. A web server or Python script usually does not, so zombies pile up.

Second, signals. The kernel does not apply default signal actions to PID 1. If PID 1 has not installed a handler for SIGTERM, the signal is simply ignored. docker stop sends SIGTERM, waits the grace period (10 seconds by default), then sends SIGKILL. The result is slow shutdowns and requests cut off without cleanup.

Fixes:

• Run a tiny init as PID 1. docker run --init adds one, and tini is the common choice.
• Make sure the app handles SIGTERM and shuts down cleanly.
• Use the exec form of the command in a Dockerfile (CMD ["node", "server.js"]). The shell form wraps the app in sh -c, so the shell becomes PID 1 and does not forward signals.

A process in D state is waiting inside the kernel for something, usually disk or network storage I/O, in a code path that cannot be interrupted by signals, so even kill -9 has no effect until the wait ends.

The reason for the design is safety. If the kernel is halfway through a disk operation and a signal tore the process out, kernel data structures could be left inconsistent. So for those short critical waits it ignores signals. Normally the wait lasts microseconds and nobody notices.

It becomes visible when the I/O never completes:

• A dying disk or a stuck RAID controller.
• A hung NFS server mounted with the default hard option, which retries forever.
• A broken multipath or SAN path.

Remember that a process in D state counts towards the load average, even though it uses no CPU. That is why a server can show a load of 50 with idle CPUs.

Many modern kernel paths use a "killable" variant of the wait, so a fatal signal can end the process, but not all paths do. Do not try to kill the process. Fix the thing it is waiting on: restore the NFS server or network, replace the disk, or as a last resort reboot. ps -eo pid,stat,wchan:30,cmd shows which kernel function each stuck process is waiting in, which usually points straight at the subsystem.

Load average is the average number of tasks that are either running on a CPU or waiting to run (or, on Linux, stuck in uninterruptible I/O wait), measured over the last 1, 5 and 15 minutes.

The three numbers from uptime are exponentially damped moving averages. Compare them to the number of CPU cores (nproc):

• Load of 8 on 8 cores means the CPUs are fully used but nothing is waiting.
• Load of 8 on 64 cores means the machine is mostly idle.
• Load of 8 on 2 cores means tasks are queuing, and users will feel slowness.

The trend matters too. If the 1-minute number is much higher than the 15-minute number, load is rising. If it is lower, the problem is easing.

The common trap is assuming load means CPU. On Linux it also includes tasks in D state, so a slow disk raises load with no CPU use. Always follow up with vmstat 1 or top to see whether the load is CPU demand (r column and us/sy) or blocked I/O (b column and wa). Brendan Gregg's article on the history of Linux load averages explains why Linux counts D state tasks.

Linux Load Averages: Solving the Mystery

A high load with idle CPUs almost always means many tasks are blocked in D state waiting on I/O, usually a slow or hung disk or network filesystem, rather than competing for CPU.

Linux counts uninterruptible sleepers in the load average, so 40 tasks stuck on a bad disk look the same as 40 busy CPU hogs in that one number.

Work through it in this order:

What to look for:

• A large b count and a high wa in vmstat confirm blocked I/O.
• A device with a very high await or a growing aqu-sz in iostat shows which disk is slow.
• dmesg lines like "task blocked for more than 120 seconds" or "nfs: server not responding" name the culprit.

Fix the storage problem and the load drops on its own. Killing the waiting processes will not help, and they cannot be killed anyway.

A context switch is when the CPU stops running one task and starts another, which means the kernel saves the first task's registers and state and loads the second's.

• A voluntary switch happens when the task gives up the CPU itself, for example because it is waiting for disk or network input, a lock, or a sleep timer.
• An involuntary switch happens when the kernel takes the CPU away because the task used up its time slice or a higher-priority task became runnable.

The split tells you something. Lots of voluntary switches mean the task is often waiting for something else, which is typical of I/O-bound or lock-heavy code. Lots of involuntary switches mean tasks are competing for CPU, so the machine is short of CPU or has too many runnable threads.

Switches are not free. Each one costs a little time, and it can throw away useful data in the CPU caches. Tens of thousands per second is normal on a busy server. A sudden jump, or a very high rate from one process, is a hint to look for too many threads or heavy lock contention.

For normal tasks, the Linux scheduler tries to give every runnable task a fair share of CPU time, tracking how much each has received and picking the one that has received the least relative to its weight.

For many years this was the Completely Fair Scheduler (CFS). Each task has a vruntime, which is the CPU time it has used, scaled by its priority weight. The scheduler keeps runnable tasks sorted by vruntime and always runs the one with the smallest value. A task that has been sleeping has a low vruntime, so it runs soon after waking. This is why interactive tasks feel responsive.

Since kernel 6.6 the default fair scheduler is EEVDF, which keeps the same idea of fair shares but also considers how soon each task needs to run, to improve latency. For an interview, the core point is the same: fairness weighted by priority.

Key knobs:

• nice values (-20 to 19) change a task's weight. A lower nice value gets a bigger share.
• Control groups (cgroups) divide CPU between groups of tasks, which is how containers get CPU shares.
• Real-time classes (SCHED_FIFO, SCHED_RR) sit above all normal tasks.

The common wrong answer is "round robin with fixed time slices". Time slices in the fair scheduler are dynamic and depend on how many tasks are runnable.

nice adjusts how big a share of CPU a normal task gets, while real-time policies (SCHED_FIFO and SCHED_RR) give a task strict priority over every normal task, so it runs whenever it is runnable.

With nice, a task is never starved. A nice 19 task still gets some CPU when others are busy, just a small share. Raising priority (a negative nice) needs root.

Real-time tasks have priorities 1 to 99 and always win over normal tasks:

• SCHED_FIFO runs until it blocks, yields or is preempted by a higher real-time priority. There is no time slicing among equals.
• SCHED_RR is the same but rotates between tasks of equal priority using time slices.

Real-time tasks are dangerous. A real-time task stuck in a busy loop can starve the whole machine, including the shell you would use to stop it. Linux has a safety limit, kernel.sched_rt_runtime_us, which by default stops real-time tasks from using more than 95 percent of CPU time. Use real-time priorities only for latency-critical work such as audio or trading code, and leave normal services alone.

CPU affinity pins a process to specific CPU cores, and NUMA (Non-Uniform Memory Access) means that on multi-socket servers a CPU reads its own local memory faster than memory attached to another socket.

Both are about keeping a task close to its data:

• A task that moves between cores loses its warm CPU caches each time, so pinning with taskset can help latency-sensitive workloads.
• On a NUMA machine, memory belongs to a node. A thread running on node 0 reading memory on node 1 pays extra latency and uses the interconnect between sockets. This can quietly cut throughput for databases and in-memory caches.

The classic symptom is a server with plenty of free memory in total, yet one node is exhausted so allocations spill to the remote node or even trigger reclaim on one node. Do not over-tune this. For most services the kernel's automatic placement is fine, and pinning only pays off when you have measured a cache or remote-memory problem.

VSZ is the total virtual address space a process has mapped, RSS is the part currently held in physical RAM, and PSS is RSS with shared pages divided fairly between the processes that share them.

• VSZ (virtual size) can be huge and is rarely useful. It includes reserved but untouched memory, memory-mapped files and shared libraries. A Java process can show tens of gigabytes of VSZ while using far less.
• RSS (resident set size) is what is really in RAM now. The catch is that shared pages, such as a library used by 50 processes, are counted in full in each process's RSS, so adding up RSS across processes overstates real memory use.
• PSS (proportional set size) divides each shared page by the number of processes sharing it. Adding up PSS gives a true total.

For "how much memory is this service really costing me", use PSS or look at the cgroup's memory usage. For a quick check, use RSS, and treat VSZ as nearly meaningless on its own.

Not necessarily. Linux deliberately uses spare RAM as page cache, so low "free" is normal, and the number that matters is "available".

The free command shows several columns:

• free is memory not used for anything at all. Linux tries to keep this small, because unused RAM is wasted RAM.
• buff/cache is memory holding file data and buffers. The kernel gives it back instantly when programs need it.
• available is the kernel's estimate of how much memory can be given to new work without swapping. This includes the reclaimable cache.

In the example only 312 MB is "free", but 11 GB is available, so the server is healthy.

Real memory pressure looks different: available is small, swap is being actively used (si and so in vmstat are non-zero over time), or the kernel log shows OOM kills. Alert on available and on swap activity, never on free.

The old advice of "free plus buffers plus cache" is the same idea, and available replaced it because not all cache can actually be reclaimed.

The page cache is RAM the kernel uses to keep recently read and written file data, so repeated reads come from memory instead of disk, and you should almost never drop it.

When a program calls read(), the kernel first checks the page cache. A hit costs microseconds. A miss goes to disk and then stores the data in the cache. Writes go into the cache first as "dirty" pages and are flushed to disk later by background writeback, or immediately when the program calls fsync().

The cache grows to fill spare memory and shrinks when programs need RAM, so it never causes out-of-memory problems by itself.

You can drop it, as root:

This is useful for one thing, benchmarking cold-cache disk performance. As a "fix" for a busy server it is harmful. It throws away useful data, so the next requests all hit disk and performance gets worse for a while. If memory looks full because of cache, there is no problem to solve.

An interview follow-up is how to see what is cached. Tools like vmtouch or fincore show how much of a file is in the page cache.

Swappiness (vm.swappiness) tells the kernel how willing it is to move unused program memory to swap versus dropping file cache when it needs to free RAM, and having swap used is not by itself a problem. Constant swapping is.

Anonymous memory (heap, stack) has no file to go back to, so it can only be reclaimed by writing it to swap. File cache can be dropped for free if clean. Swappiness balances the two. The default is 60, and a lower value prefers dropping cache over swapping.

What matters is swap activity, not swap usage:

• A few hundred MB of swap used, with no ongoing traffic, just means cold pages were parked there. That frees RAM for cache and is healthy.
• Continuous swap-in and swap-out is thrashing. The system spends its time moving pages, and everything slows to a crawl.

Non-zero si and so over many seconds is the warning sign.

Some workloads do want swap tuned. Databases usually run with a low swappiness so hot memory is not pushed out, while systems with no swap at all risk the OOM killer firing sooner because anonymous memory has nowhere to go. The common wrong answer is "disable swap always". It depends on the workload.

A page fault happens when a program touches a virtual address that is not currently mapped to physical RAM, and the kernel steps in to fix it. A minor fault is fixed without disk I/O, and a major fault needs to read from disk.

Page faults are a normal part of how memory works. They are not errors.

• Minor fault: the data is already in memory (for example in the page cache), or the page is brand new and only needs a zeroed page mapped in. Cost: microseconds.
• Major fault: the page must be read from disk, for example from a memory-mapped file that is not cached or from swap. Cost: milliseconds, thousands of times slower.
• Invalid access: the address is not valid at all. The kernel sends SIGSEGV, and you see a segmentation fault.

To check a process: ps -o pid,min_flt,maj_flt,cmd -p <pid> or /usr/bin/time -v ./app. A rising major fault count means the process is waiting on disk for memory, which points at swapping or cold memory-mapped data.

Linux by default lets programs allocate more memory than the machine can back, because allocations only reserve address space, and real RAM is handed out when a page is first written, so a malloc can succeed and the failure only shows up later as an OOM kill.

This is controlled by vm.overcommit_memory:

Why the kernel does this: most programs reserve far more than they touch, and fork() temporarily doubles the apparent memory of the parent. Strict accounting would refuse many harmless allocations.

The cost is that failure moves from malloc returning NULL (which a program can handle) to the OOM killer firing later (which it cannot).

Some software wants a specific mode. Redis, for example, recommends vm.overcommit_memory = 1, because it forks to save snapshots, and a strict mode could make that fork fail on a big dataset even though copy-on-write means little extra memory is really needed. Systems that must never be OOM-killed sometimes use mode 2, but then need careful sizing.

The process has hit its limit on open file descriptors, so first find the limit and the current count for that process, then raise the limit in the right place, and also check whether the application is leaking descriptors.

The error has two forms: EMFILE means this process reached its own limit, and ENFILE means the whole system reached fs.file-max.

Raising the limit depends on how the process is started:

• systemd service: set LimitNOFILE=65536 in the unit file. This is the most common place people forget.
• Login sessions: /etc/security/limits.conf, which is applied by PAM and does not affect systemd services.
• Containers: docker run --ulimit nofile=65536:65536, or the runtime's default.
• System-wide: sysctl fs.file-max.

The soft limit is the one enforced, and a process can raise it up to the hard limit.

The trap is raising the limit and calling it done. If the count keeps climbing over days, the application leaks descriptors, for instance by not closing HTTP responses or database connections. Raising the limit only delays the crash, so fix the leak.

A deleted file keeps its disk space as long as some process still has it open, because Linux frees the space only when the last link and the last open descriptor are gone.

When you run rm, you remove the name from the directory. The inode and its data blocks stay until nothing refers to the file. A service that is still writing to the old log keeps the file alive, and du no longer shows it, but df still counts the space.

Find those files with lsof:

Two ways to free the space:

• Restart or reload the process so it closes and reopens its log. That is the clean fix, and it is what logrotate signals are for.
• Truncate the open file through /proc, without restarting:

The lesson is to truncate logs (: > app.log) or rotate them properly, rather than deleting a file that a running service holds open. A symptom that df and du disagree is almost always this.

strace shows every system call a process makes, with arguments, results and optionally timings, which makes it the fastest way to see what a misbehaving program is actually doing, but it slows the target down significantly.

Common uses:

What it helps find:

• A program failing with "No such file or directory" shows the exact path it tried to open, and ENOENT in the result.
• A permissions problem shows as EACCES.
• A hang shows which call it is stuck in.
• A slow program shows which calls take the time (-T) or how many calls it makes (-c).

The risk is overhead. strace uses ptrace, which stops the target at every system call and every return. A busy process can slow down by an order of magnitude, so attaching to a production service under load can cause an outage by itself.

Safer options on a busy server are perf trace, or eBPF-based tools (bpftrace, opensnoop, execsnoop), which watch events with far lower cost. Use strace for quick diagnosis on staging, on lightly loaded processes, or with filters (-e trace=...) to cut the work.

strace manual page

Check its state first, then look at which system call it is blocked in, then map the file descriptor in that call to a real file or connection.

1. State. ps -o pid,stat,wchan:30,cmd -p <pid>. If it is in D, it is stuck on I/O in the kernel, and wchan names the kernel function. If it is in S, it is waiting for an event.
2. Where it is blocked.

3. Read the call. What the last call tells you:

4. Map the descriptor. ls -l /proc/5120/fd/7 shows the file, or a socket:[inode], which ss -tanp can match to a remote address.

For a multi-threaded process, look at each thread (ls /proc/5120/task), since only one thread may be stuck. A thread waiting in futex for ever while others idle usually means a deadlock, so take a thread dump (for example jstack on Java) to see who holds the lock.

The usual culprits are inode exhaustion, a deleted file still held open, or reserved blocks, and df -i is the first check.

1. Out of inodes. Millions of tiny files (session files, mail queues, cache directories) can use up every inode before the space runs out.

Fix: delete the files, or move them. On ext4 the inode count is fixed at format time, while XFS allocates inodes dynamically and rarely hits this.

2. Deleted but open files. Space is not released until the last process closes the file (lsof +L1). In this case df is high rather than low, but the symptom is similar when logs fill the disk.

3. Reserved blocks. ext4 reserves around 5 percent of space for root by default. Normal users get "no space" while df still shows some available to root. It can be tuned with tune2fs -m.

4. Quota or a different layer. A user or project quota, a full container overlay layer, or a mount that is smaller than you think.

The wrong move is to start deleting large files blindly. Check df -h and df -i first, because the fix is different for each cause.

A normal write() only copies data into the kernel's page cache and returns, so it is fast but not durable, fsync() forces that data to the storage device, and direct I/O (O_DIRECT) bypasses the page cache entirely.

• Buffered I/O (default). write() returns as soon as data is in the page cache. The kernel flushes dirty pages in the background, usually within about 30 seconds. If the machine loses power before that, the data is gone, even though the program thought it was written.
• fsync() / fdatasync(). Blocks until the file's data (and for fsync, its metadata) has reached stable storage. It is the way to get durability, and it is slow because it waits for the device.
• Direct I/O. Skips the cache. Databases that manage their own buffer pool use it to avoid caching the same data twice.

This is why databases are careful. A database must not tell a client "committed" before the commit record is safely on disk, so PostgreSQL calls fsync on its write-ahead log at commit. The speed of fsync often decides how many transactions per second a database can do, which is why fast storage with a protected write cache helps so much.

A trap: fsync on a file does not guarantee that a newly created file's directory entry is durable. For that you also fsync the directory.

The writeback behaviour is tunable with vm.dirty_background_ratio and vm.dirty_ratio. Large dirty buffers can lead to a burst of flushing that stalls writers, a classic cause of periodic latency spikes.

iowait is the percentage of time a CPU was idle while at least one task on the system was waiting for disk I/O, which makes it a measure of idle time, not of work, and not a direct measure of how slow the disk is.

The common misreading is "iowait is the CPU waiting for the disk, so CPU is wasted". In fact, a CPU in iowait has nothing else to run. It is idle. If other tasks wanted to run, the CPU would run them, and the time would not count as iowait.

Consequences:

• High wa means that tasks are blocked on I/O and the CPUs have spare capacity. It points toward storage, not a CPU shortage.
• Low wa does not prove the disk is healthy. If the CPUs are busy with other work, a saturated disk can show almost no iowait.
• The number depends on how many CPUs there are. One blocked task on a 64-core machine barely moves the average.

So use it as a hint, then confirm with device-level numbers:

An interviewer may follow up with "what else counts as I/O wait?". Remember that tasks blocked in D state also raise the load average, whether or not iowait is high.

Look at latency (await) and queue size (aqu-sz) first, then throughput, and treat %util with care, because it only shows how often the device had at least one request in flight, not how close it is to its limit.

Key columns in recent sysstat versions:

How to judge:

• A rising await with a growing aqu-sz means requests are queuing, so the device cannot keep up. This is the real sign of saturation.
• A spinning disk with %util near 100 is genuinely saturated, because it handles roughly one request at a time.
• An SSD or NVMe drive handles many requests in parallel. It can show 100 percent util while having lots of spare capacity, because one request was always in flight. Same for RAID arrays and cloud volumes.

Compare await with what is normal for that storage: around a millisecond for NVMe, a few milliseconds for a cloud SSD, and tens of milliseconds for a busy spinning disk. A sudden jump matters more than the absolute number. In cloud environments also check whether the volume hit its provisioned IOPS or throughput cap, which shows up as flat throughput with rising latency.

An I/O scheduler decides the order in which block requests are sent to a disk, and on modern kernels the choice is usually none for NVMe and fast SSDs, and mq-deadline for most other devices.

Modern kernels use the multi-queue block layer, which offers these options:

Why reordering matters: on a spinning disk, sorting requests by position reduces head movement. On NVMe there is no seek penalty, so reordering is wasted work, and none is fastest.

To make it persistent, use a udev rule or a boot parameter. In virtual machines the guest often sees a virtual disk where the real scheduling happens on the host, so the guest setting matters less. An honest answer in an interview is that defaults are good for most workloads, and you should only change the scheduler after measuring a problem.

vmstat gives a one-line-per-second summary of CPU, memory, swap and I/O, and the most useful columns are r, b, si, so, wa and cs.

Remember that the first line is an average since boot, so ignore it and read the lines after it.

Quick readings:

• High r, high us: CPU-bound application.
• High sy: lots of kernel work, such as syscalls, networking or context switching.
• High b, high wa: storage is the bottleneck.
• Non-zero si/so: the machine is short of RAM.
• High st: a noisy neighbour on a virtual machine.

epoll is a Linux mechanism that lets one thread watch thousands of file descriptors and be told only about the ones that are ready, whereas select and poll must scan the whole list on every call.

With select or poll, you hand the kernel the full list of descriptors each time, and the kernel checks all of them, so the cost grows with the number of connections, even when only a few are active. With 50,000 mostly idle connections, that is a lot of wasted work.

epoll splits this into steps:

1. epoll_create makes an epoll instance in the kernel.
2. epoll_ctl adds or removes descriptors once, and the kernel remembers them.
3. epoll_wait sleeps until something is ready and returns only the ready descriptors.

The cost now depends on the number of ready events, not the number of watched connections.

It supports two modes. Level-triggered keeps reporting a descriptor as long as it is ready, which is simple and safe. Edge-triggered reports only when the state changes, which is faster but needs careful code that reads until the call would block.

This is the foundation of event-driven servers such as nginx, Node.js (through libuv), Redis and HAProxy. It explains how a single-threaded server can handle huge numbers of connections, as long as the work per event is small.

epoll(7) manual page

Namespaces limit what a process can see, cgroups limit how much it can use, and a container is just an ordinary process started with both, plus some security filtering.

Namespaces give a process its own private view of a system resource:

Cgroups (control groups) account for and limit resources: CPU time (cpu.max), memory (memory.max), block I/O (io.max) and number of processes (pids.max).

On top of these, runtimes add Linux capabilities, seccomp system call filters and AppArmor or SELinux profiles to reduce what the process can do.

Important consequence: there is no separate kernel or hypervisor. A container shares the host kernel, so it starts in milliseconds, but a kernel bug can affect every container on the host.

You can see this directly:

A good interview point is that docker run does nothing magic. It calls clone() with namespace flags and writes limits into the cgroup filesystem.

perf is the Linux profiler that samples what the CPUs are executing, and a flame graph is a way to draw those samples so that the most expensive code paths stand out as the widest boxes.

Collect samples, for example 99 per second for 30 seconds, with stack traces:

How to read the picture:

• The horizontal width of a box is the share of samples that included that function. Wider means more CPU time. The x-axis is not time, and the left-to-right order is alphabetical, not chronological.
• The vertical axis is call stack depth. The bottom is the entry point, and each box above is a function it called.
• Look for wide boxes at the top edge (plateaus). Those are the functions actually burning CPU, rather than just calling others.

Practical tips: make sure binaries keep frame pointers or use DWARF unwinding, or the stacks come out broken. Language runtimes need symbol support (for example a perf map for Java or Node). Sample for long enough to be representative.

Flame graphs answer "where is the CPU time going". For waiting time rather than CPU, use off-CPU analysis (tools such as offcputime from bcc).

Brendan Gregg: Flame Graphs

CPU limits are enforced with a quota in short periods, so a multi-threaded container can burn its whole quota in a few milliseconds and then be frozen (throttled) for the rest of the period, which shows up as latency spikes even though the average usage looks low.

In cgroup v2 the limit is cpu.max, for example 50000 100000. That means 50 ms of CPU time per 100 ms period, which is half a core. If the app has 8 busy threads running in parallel, they use 8 ms of CPU for every 1 ms of wall time. The 50 ms quota is gone after about 6 ms, and every thread is paused for the remaining 94 ms of the period. A request that arrives during the pause waits.

Averages hide this, because 100 ms of throttling every few periods still averages out to a low usage over a minute.

How to confirm it:

In Kubernetes the same data shows up as the container_cpu_cfs_throttled_periods_total metric.

Fixes, roughly from best to worst:

• Raise the limit, or remove the CPU limit and keep only a request, so the pod can use idle CPU on the node.
• Match the thread count to the limit (for example set GOMAXPROCS in Go, or the worker count in a pool). Many runtimes see all node cores, not the cgroup limit.
• Spread the load over more replicas.

The trap is assuming low average CPU means no CPU problem. Look at throttling counters, not just utilisation.

cgroup v2 documentation

When the kernel cannot free enough memory, the OOM killer scores every process by how much memory it uses, adjusts that by oom_score_adj, and kills the one with the highest score.

The score is mostly the process's memory footprint (resident memory, swap and page tables) as a share of available memory. /proc/<pid>/oom_score_adj ranges from -1000 to 1000 and shifts it. A value of -1000 makes a process immune, and a positive value makes it a preferred victim. That is how critical daemons such as sshd are protected, and how Kubernetes ranks pods by quality of service class.

To investigate after the fact:

The log lines show which process died, its memory usage, and a list of all candidates with their sizes.

Two kinds of OOM matter, and mixing them up is the classic error:

• Host OOM: the whole machine ran out of memory, and the killer picks across all processes.
• cgroup OOM: a container or service hit its own memory limit (memory.max). The kill happens inside that cgroup, even if the host has plenty of RAM free. In Docker and Kubernetes this shows as exit code 137 (128 plus signal 9) and the reason OOMKilled.

Fixes depend on the cause. Find the process that grew (leak or load), raise the limit if it is simply too low, or set oom_score_adj to protect the important process. Do not just add swap and hope.

First separate what kind of memory is growing: the process's own anonymous memory (heap), file cache attributed to it, or shared memory, because only the first is a classic leak.

A practical approach:

1. Confirm the trend. Sample resident memory over time rather than guessing from one number.

2. Break it down. smaps_rollup splits memory into private and shared, and into anonymous and file-backed.

3. Interpret.

   • Growing Anonymous memory with load that does not grow suggests a leak or unbounded in-process cache.
   • Growing file-backed memory is usually mapped files or page cache, which the kernel can reclaim, so it is not a leak.
   • A big fixed-size jump after startup, then flat, is just warm-up.

4. Find the code. Use the language's own tools, such as heap profilers and pprof for Go, heap dumps for Java, tracemalloc for Python, or valgrind and heaptrack for native code.

Two traps are worth naming. Allocators such as glibc malloc may hold freed memory instead of returning it to the OS, so RSS stays high with no leak (fragmentation). And a container's memory usage metric usually includes page cache, so look at the working set rather than the raw usage number before deciding it is a leak.

When Redis forks a child to write a snapshot, parent and child share all memory pages using copy-on-write, so memory use only rises as the parent keeps writing, and the fork itself pauses the parent briefly in proportion to how much memory it has mapped.

Two separate effects:

• Fork latency. fork() does not copy data, but it must copy the page tables, and those grow with memory size. On a very large instance, especially a virtual machine with slow page-table handling, the call can block the main thread for tens or hundreds of milliseconds. That is a visible latency spike.
• Copy-on-write memory growth. While the child writes the snapshot, every page the parent modifies must first be copied, so the child keeps its consistent view. A write-heavy workload can touch a large share of pages during the snapshot, and memory can approach double the dataset size in the worst case.

Transparent Huge Pages make it worse. With 2 MiB pages, changing one byte copies 2 MiB instead of 4 KiB, which is why Redis advises disabling THP.

Mitigations:

• Leave headroom for copy-on-write memory, rather than sizing the instance to the dataset.
• Set vm.overcommit_memory = 1 so the fork is not refused.
• Disable THP.
• Take snapshots on a replica rather than the primary.

This question is really testing whether you understand that fork() is cheap in copying but not free in page-table work, and that copy-on-write is paid for gradually.

Transparent Huge Pages (THP) let the kernel back memory with 2 MiB pages instead of 4 KiB pages automatically, which reduces page-table overhead but can cause latency spikes, so many databases recommend disabling or limiting them.

Bigger pages mean fewer entries in the page table and in the CPU's translation cache (TLB), so memory-hungry applications can run a little faster. The kernel hands out huge pages without the application asking, and a background thread (khugepaged) merges small pages into huge ones later.

The problems for latency-sensitive services:

• Compaction stalls. To find 2 MiB of continuous physical memory, the kernel may have to move pages around, and the allocation waits while it does.
• Copy-on-write cost. Writing to a shared huge page copies 2 MiB, as in the Redis fork example.
• Memory bloat. A sparse allocation can be rounded up to a whole huge page.

madvise is a good middle ground. Note that this change does not survive a reboot unless you set it in a boot parameter or a systemd unit. Check the documentation of your database before changing it, since advice differs. Explicit huge pages (hugetlbfs) are a different feature, reserved up front and not affected by these problems.

Work from the device up to the process: confirm the device is slow, identify the process generating the I/O, and check the hardware and cloud limits to decide whether it is load or a fault.

1. Confirm and measure at the device.

2. Find the culprit process.

3. See the latency distribution. Averages hide outliers, and eBPF tools from the bcc collection give histograms:

4. Check whether the hardware is the problem.

5. Check external limits. On cloud volumes, compare IOPS and throughput with the provisioned cap. Flat throughput and rising queue depth means you hit the cap, not a failure. On RAID or SAN, check the controller, cache battery and rebuild status. A degraded array or a rebuild causes heavy latency.

How to decide: if one process is generating far more I/O than usual (a backup, a log flood, a runaway query), it is a load problem, and the fix is to throttle or reschedule it (ionice, cgroup io.max). If latency is high with low load, and dmesg shows errors, the device is failing and needs replacing.

A good answer shows method, not just tools: device first, then process, then hardware, then limits.

Use a fixed checklist that looks at every resource (CPU, memory, disk, network) for utilisation, saturation and errors, instead of guessing, so you find the bottleneck quickly and do not miss one.

This is the idea of the USE method: for each resource, check Utilisation, Saturation and Errors. A first-minute sequence of commands covers it:

Once one resource stands out, go deeper with a targeted tool (perf, strace, biolatency, ss -ti). Mention that you also check what changed recently, such as a deploy or a config change, and compare with a baseline, because "slow" only means something relative to normal.

Brendan Gregg: Linux Performance

High %sy means the CPUs are spending their time inside the kernel, so find which kind of kernel work it is (system calls, interrupts, networking, context switching or page faults) and which process causes it.

1. Break it down per CPU and per type.

High %soft on one core often means network packet processing is bottlenecked on a single CPU.

2. Find the process.

3. Profile the kernel. perf shows where kernel time goes:

Typical findings and fixes:

A good answer does not stop at "use top". It narrows the problem in steps, and ends with a profile that names the function.

Steal time is the share of time that a virtual CPU was ready to run but the hypervisor gave the physical CPU to another virtual machine instead, so high %st means your VM is being starved by the host, not by your own workload.

You see it as st in top, vmstat and mpstat. Inside the guest the symptoms look like a slow CPU: tasks are runnable, but they make less progress than the CPU usage suggests.

Typical causes:

• Noisy neighbour. Other VMs on the same physical host are busy.
• Oversubscription. The provider allocated more virtual CPUs than physical cores.
• Burstable instances. On types such as the AWS T-series, when CPU credits run out the instance is throttled to a baseline.
• Small allocation. A VM given a fractional share of CPU.

How to respond:

1. Confirm it is consistent, not a blip: mpstat -P ALL 1, and compare over time.
2. Rule out burst credits. Check the cloud provider's credit balance metric.
3. Mitigate. Stop and start the instance (it often lands on a different host), move to a larger or dedicated instance type, or switch from burstable to a fixed-performance family.
4. Raise it with the provider if it persists, with the timestamps.

There is nothing to fix in the guest itself. Tuning your own software will not help, so the answer is to change where or on what the VM runs. The trap is chasing the application when the stolen time shows the host is the cause.

With the default hard mount option, an NFS client retries a failed request forever, so every process that touches the mount blocks in D state until the server answers, and cannot be killed until then.

This is deliberate. NFS hard mounts protect data integrity: the application never sees an I/O error for a transient outage, it just waits. The cost is that one dead server can freeze many processes, push the load average very high, and make even ls or df on the mount hang.

Diagnose:

Recover, in order of preference:

1. Fix the server or the network path. Once it answers, the waiting processes continue on their own.
2. If the server is gone for good, unmount: umount -f /mnt/data, or lazy unmount umount -l /mnt/data to detach the mount from the tree, while stuck processes may stay stuck.
3. As a last resort, reboot the client.

Prevention is a design choice. soft mounts return an error after retries (timeo, retrans), so applications fail fast instead of hanging, but they risk silent data corruption on writes, so they suit read-only or non-critical data. Put critical paths on local storage where possible, keep application health checks that do not touch the mount, and use timeouts around anything that might block on it.

eBPF lets you load small programs into the running kernel that attach to events (system calls, kernel functions, tracepoints, network packets) and process them in place, so you get deep visibility with low overhead and without changing or stopping the target.

Why it is safe: before a program is accepted, the kernel's verifier checks it. It must terminate (no unbounded loops), only access memory in permitted ways, and only call approved helper functions. A buggy eBPF program is rejected at load time instead of crashing the kernel.

Why it is cheap: the filtering and aggregation happen inside the kernel. A tool can count events or build a latency histogram in a kernel map and send only the summary to user space. By comparison, strace uses ptrace, which stops the target twice per system call and copies every event out, which can slow a busy process dramatically.

Common tools:

Limits to mention: it needs a reasonably recent kernel and root (or CAP_BPF), the output of some probes depends on kernel version, and poorly designed probes on very hot events can still add overhead. Many observability products (Cilium, Pixie, Parca) use eBPF underneath.