Linux Interview Questions

A hard link is a second directory entry that points at the same inode as the original file, so both names share identical data, permissions, and ownership, and the data stays on disk until every hard link to it is removed. A symbolic link (symlink) is its own small file that just stores a path string pointing at the target, so it can cross filesystems and even point at something that doesn't exist, but it breaks if the target moves or is deleted.

The common wrong answer treats a symlink as "just a shortcut" with no real distinction from a hard link. The actual test an interviewer is checking for is the inode: hard links share one inode and one set of data blocks; symlinks get their own inode holding a path.

ln(1) man page

It sets read, write, and execute for the file's owner, and read plus execute (no write) for the group and everyone else. Each digit is a sum of read (4), write (2), and execute (1): 7 = 4+2+1 for the owner, 5 = 4+1 for group and others.

This comes up because candidates often memorize "755 means executable" without being able to derive it, which falls apart the moment they're asked for a less common mode like 640 or 700. Knowing the arithmetic means you can read or set any permission on the spot instead of recalling a table.

chmod(1) man page

A process has its own isolated memory space, file descriptors, and address space; the kernel schedules it independently and one process crashing doesn't directly corrupt another's memory. A thread is a unit of execution within a process that shares that process's memory and open files with its sibling threads, so threads are cheaper to create and communicate through shared memory, but a bug in one thread (like writing to bad memory) can crash the whole process.

On Linux specifically, both are actually created with the same underlying clone() system call — a thread is really just a process that shares more resources with its parent than a normal child process does. That's worth mentioning because it's the detail that shows real familiarity with the kernel rather than a textbook definition.

clone(2) man page

/etc/fstab tells the system which filesystems to mount automatically at boot, and where. Each line describes one filesystem: the device or UUID, the mount point, the filesystem type, mount options, and two numbers used for dump backup and fsck ordering.

Editing it by hand is risky because a typo can leave a server unable to boot into a full environment — that's why the standard advice is to test a new entry with mount -a before rebooting, so a mistake surfaces as an error message rather than a boot failure.

fstab(5) man page

A cron job is a task the cron daemon runs automatically on a schedule you define, instead of you running it by hand. You define schedules with crontab -e for a per-user crontab, or by dropping a file in /etc/cron.d/; the schedule is five fields for minute, hour, day of month, month, and day of week, followed by the command to run.

A detail worth knowing: cron jobs run with a minimal environment (no login shell, often no PATH beyond the basics), which is the single most common reason a script that "works fine when I run it manually" silently fails under cron — always use absolute paths and set PATH explicitly inside cron scripts.

crontab(5) man page

dpkg and rpm are low-level package managers that install, remove, and query individual .deb or .rpm package files, but they don't resolve dependencies or talk to a remote repository — if a package needs a library that isn't installed, dpkg/rpm will just fail. apt (Debian/Ubuntu) and yum/dnf (RHEL/Fedora/CentOS) are the higher-level tools built on top of them: they fetch packages from configured repositories, work out the dependency graph, and hand the actual install off to dpkg or rpm underneath.

The practical reason this distinction matters: if you're ever handed a lone .deb file with no repository behind it, apt install ./package.deb (or plain dpkg -i followed by apt --fix-broken install to pull in missing dependencies) is the right tool, not trying to add it to a repo you don't need.

Debian package management docs

It searches recursively (-r) through every file under the current directory for lines containing the text TODO, printing each matching line along with the file it was found in. grep is a pattern-search tool, and its most-used flags are worth knowing cold: -i ignores case, -v inverts the match (shows lines that don't match), -n prints line numbers, -l prints only filenames instead of the matching lines, and -E enables extended regular expressions so you can use +, ?, and | without escaping them.

grep(1) man page

/etc/passwd holds one line per user account: username, UID, GID, home directory, default shell, and so on — it's world-readable because plenty of ordinary tools need to look up a username or home directory. /etc/shadow holds the actual password hashes plus password-aging policy, and it's readable only by root, which is the entire reason it exists as a separate file: keeping hashes out of a world-readable file makes offline password-cracking attempts much harder, since an unprivileged user can't even read the hashes to attack them.

shadow(5) man page

> redirects a command's output into a file, overwriting whatever was already there. >> redirects output and appends it to the end of the file, leaving existing content intact.

The mistake this catches in practice: running a script with > inside a loop or a cron job, where each run wipes out everything the previous run wrote instead of accumulating a log — using >> is almost always what you actually want for anything that appends to a running log file.

Bash manual — Redirections

nice starts a new process with a given scheduling priority, and renice changes the priority of a process that's already running. The value ranges from -20 (highest priority, most favored by the scheduler) to 19 (lowest priority, most willing to yield CPU time to other processes) — the name comes from how "nice" the process is being to everything else competing for CPU.

Only root can set a negative value (raise a process's priority above the default); any user can lower their own process's priority. This matters in practice for running a background batch job — like a backup or a big compile — without it competing for CPU with interactive or latency-sensitive workloads on the same box.

nice(1) man page

A Physical Volume (PV) is a raw disk or partition that's been initialized for LVM use. A Volume Group (VG) pools one or more PVs together into a single block of storage, hiding the fact that it might span several physical disks. A Logical Volume (LV) is a chunk carved out of that pool that you actually format with a filesystem and mount — it's the LVM equivalent of a traditional partition, except it can be resized or moved without caring which physical disk its bytes happen to live on.

The layering is the thing worth being able to draw: PV → VG → LV. A single VG can span multiple physical disks, and a single LV can be extended live as long as its VG has free space, which is the entire practical value of LVM over plain partitioning.

Red Hat — LVM Administration

No — apt update only refreshes the local index of what packages and versions are available in the configured repositories; it doesn't install or change anything on the system. apt upgrade is the separate command that actually installs newer versions of already-installed packages, using the index that update just refreshed.

This trips people up because the names sound like they should do the same thing, and running only apt update and assuming the system is now patched is a genuinely common and dangerous mistake — the index is current, but nothing has actually been installed. The correct sequence for applying patches is always both commands, in order.

Debian apt(8) man page

PATH is a colon-separated list of directories the shell searches, in order, when you type a command name without a full path — it's how typing ls finds /usr/bin/ls without you spelling out the whole path every time. "Command not found" despite the file existing almost always means the file's directory isn't in PATH, the file isn't marked executable (chmod +x), or — a subtler case — it's a script missing a valid shebang line (#!/usr/bin/env bash) telling the kernel which interpreter to run it with.

A related trap worth naming: the current directory (.) is deliberately not in PATH by default on most distros, specifically as a security measure — without that, a malicious script named ls dropped into a directory you cd into could silently run instead of the real /bin/ls.

Bash manual — PATH

The three numbers are the average number of processes wanting CPU time (running or waiting on I/O) over the last 1, 5, and 15 minutes respectively. Comparing them tells you the trend: if the 1-minute number is much higher than the 15-minute number, load is spiking right now; if it's the other way around, load is settling down after an earlier spike.

A load average number on its own is meaningless without knowing how many CPU cores the box has — a load of 4 is idle on a 32-core server and badly overloaded on a 2-core one, so always check nproc alongside it.

Brendan Gregg — Linux Load Averages

/bin and /sbin traditionally held essential binaries needed even before /usr was mounted, with /sbin reserved for system administration tools typically run by root. /usr/bin holds the bulk of regular user-facing programs installed by the distro's package manager, and /usr/local/bin is for software installed manually or outside the package manager, kept separate so it doesn't get overwritten by a system update.

On most modern distros /bin and /sbin are now just symlinks to their /usr equivalents (the "usr merge"), so the historical distinction has mostly collapsed — but /usr/local/bin still matters in practice as the conventional place to put a script or binary you built or downloaded yourself, kept out of the package manager's way.

Filesystem Hierarchy Standard

/proc is a virtual filesystem — it doesn't contain real files on disk, it's the kernel exposing live information about the system and running processes as if they were files, generated on the fly when you read them. /proc/cpuinfo shows detailed CPU information, /proc/meminfo shows memory statistics, and /proc/<PID>/ holds a directory per running process with details like its open file descriptors (/proc/<PID>/fd/), environment variables (/proc/<PID>/environ), and current working directory.

A concrete practical example already covered elsewhere in this topic: /proc/<PID>/fd/ is exactly how you inspect or truncate a deleted-but-still-open file that a running process is holding onto — there's no other direct way to reach an open file descriptor once its directory entry is gone.

proc(5) man page

Installing from a repository gets you a pre-compiled binary that the distro's maintainers have already built, tested, and packaged with sane defaults, and the package manager tracks it for easy upgrades and clean removal later. Building from source means compiling the code yourself with ./configure && make && make install (or an equivalent toolchain), which the package manager knows nothing about — there's no automatic upgrade path and no clean uninstall unless you're careful to track what got installed where.

The repository is the right default almost always. Building from source is worth the extra effort mainly when you need a version newer than what the distro ships, need to enable a compile-time option the packaged build doesn't include, or are working with genuinely niche software that never got packaged for your distro at all.

Debian New Maintainers' Guide

Setuid on an executable makes it run with the file owner's privileges rather than the caller's — passwd is the classic example, since it needs root to edit /etc/shadow on behalf of an ordinary user. Setgid on an executable does the same for the group; on a directory, it makes new files inside inherit the directory's group instead of the creator's primary group, which is why shared team directories use it. The sticky bit on a directory (like /tmp) stops users from deleting or renaming files they don't own, even if they have write access to the directory itself.

The trap here is assuming setuid on a directory works like setgid on a directory — it doesn't; setuid on a directory has no defined effect on Linux. Interviewers ask this because a stray setuid bit on the wrong binary is a real privilege-escalation vector, so knowing what each bit does (and where it's dangerous) matters more than memorizing the octal digit.

Special permissions — Red Hat docs

kill -15 (SIGTERM) is a polite request asking the process to shut down; the process can catch that signal and clean up — closing files, flushing buffers, releasing locks — before it exits. kill -9 (SIGKILL) can't be caught, blocked, or ignored; the kernel terminates the process immediately with no chance to clean up.

The reason interviewers ask this is to see whether you reach for -9 by default, which is the common wrong answer. SIGKILL is a last resort: a database killed with -9 mid-write can leave corrupted files or orphaned locks, whereas SIGTERM gives it a chance to commit or roll back first. The right sequence is SIGTERM, wait a few seconds, check if the process is gone, and only escalate to SIGKILL if it's still hung.

signal(7) man page

Load average counts processes that are either running on the CPU or waiting in an uninterruptible state, most commonly blocked on disk I/O. So a high load average with idle-looking CPU almost always points at I/O wait, not compute — something is stuck waiting on a slow disk, a hung NFS mount, or a saturated storage backend, not waiting for CPU time.

To confirm, check the wa (I/O wait) column in top or vmstat, and look for processes in D state (uninterruptible sleep) with ps aux | grep " D". A process stuck in D state can't even be killed with SIGKILL until the I/O it's waiting on completes, which is itself a useful thing to know when someone asks why kill -9 "isn't working."

The common wrong answer is treating load average as a pure CPU metric and immediately looking for a runaway process to kill — that's the right instinct for high CPU usage, but the wrong diagnosis here.

Brendan Gregg — Linux Load Averages

df reports free and used space at the filesystem level, reading it straight from the filesystem's block allocation. du walks a directory tree and adds up the size of the files it can see, which is a completely different calculation. They usually roughly agree, but they diverge whenever something exists that du can't or won't count: a large deleted-but-still-open file (the space is allocated at the filesystem level but invisible to a directory walk), sparse files, or mount points nested inside other mount points that du doesn't cross into by default.

The scenario worth knowing cold is "df says the disk is full, du -sh /* doesn't add up to the total" — that's almost always a deleted file still held open by a running process (see the lsof +L1 answer above), not a du bug.

GNU coreutils — du and df

Firmware (BIOS or UEFI) runs first and does hardware initialization, then hands off to a bootloader — GRUB on most modern distros — which loads the Linux kernel and an initial RAM filesystem (initramfs) into memory. The kernel takes over, initializes hardware drivers, and mounts the initramfs as a temporary root filesystem, which contains just enough (drivers, tools) to find and mount the real root filesystem, often because that root lives on an encrypted or LVM volume that needs extra setup before it's mountable. Once the real root is mounted, the kernel starts PID 1 — on virtually every modern distro that's systemd — which then brings up targets (systemd's equivalent of old-style runlevels): mounting remaining filesystems, starting networking, starting services in dependency order, and finally starting a login manager or getty on the console.

The detail that separates a real answer from a memorized one is initramfs: knowing why it exists (the kernel needs drivers to reach the real root, but can't load those drivers from the real root before it can reach it) rather than just naming it as a step in the sequence.

systemd bootup man page

Start broad, then narrow. systemctl status <service> gives the current state and the last handful of log lines in one place, which is usually enough to see the immediate error. If it's not, journalctl -u <service> shows the full log for that unit, and journalctl -u <service> -b scopes it to the current boot, which matters on a long-running box where you don't want last month's noise. If the service depends on other units, systemctl list-dependencies <service> shows what it's waiting on, and a failed dependency further up the chain is a very common reason a service "fails" with a misleading error in its own logs.

The trap: reading only systemctl status's truncated log snippet and guessing at the cause, instead of pulling the full unit log with journalctl -u, which usually has the actual stack trace or exit code that status cuts off.

systemctl man page

TCP is connection-oriented: it establishes a handshake, guarantees delivery and ordering, and retransmits lost packets, at the cost of extra overhead and latency. UDP is connectionless: it sends packets with no guarantee of delivery, ordering, or duplicate protection, but with much lower overhead since there's no handshake or retransmission logic.

Choose TCP when correctness matters more than speed and you can tolerate a bit of latency — HTTP, database connections, file transfers, anything where a dropped or reordered packet would corrupt the result. Choose UDP when speed and low latency matter more than perfect delivery, or when the application layer already handles reliability itself — DNS lookups, VoIP and video calls (a dropped packet just means a tiny glitch, not worth the delay of retransmitting it), and game state updates where a stale retransmitted packet is worse than a dropped one.

RFC 793 — TCP · RFC 768 — UDP

Chain small, streaming Unix tools instead of reading the file into a program's memory. grep filters to just the error lines as it streams through the file, sort orders them so identical lines end up adjacent, uniq -c collapses adjacent duplicates into a count, and a second sort -rn orders those counts descending; head trims it to the top 10. Every one of these tools processes its input as a stream rather than loading the whole file at once, which is exactly why this pipeline scales to files far bigger than available RAM — the "load it into memory" instinct (like reading the whole file into a Python list) is the wrong default for anything multi-gigabyte on a constrained box.

The subtlety worth mentioning: uniq -c only collapses adjacent duplicate lines, which is why the first sort has to happen before it — without that sort, identical error messages scattered throughout the file wouldn't get grouped together and the counts would be wrong.

GNU coreutils — uniq

su switches you to another user's shell entirely — typically root — and from that point on every command runs as that user until you exit, and it requires knowing the target account's own password. sudo runs a single specified command with elevated privileges and then returns you to your own shell, authenticating with your password (or none, depending on config), and every invocation is logged with the username and exact command that ran.

That logging is the real reason sudo won by default on most modern distros: with su root, an audit log just shows "root did X" with no record of which human was behind the keyboard, whereas sudo gives you an accountable trail per person. sudo access is also controlled per-user and per-command through /etc/sudoers (or drop-in files under /etc/sudoers.d/), so you can grant someone permission to restart one specific service as root without handing them the root password at all.

sudoers(5) man page

export FOO=bar typed directly in a shell only exists for that shell session and any child processes it spawns — close the terminal and it's gone. Putting it in ~/.bashrc makes it available in every new interactive bash shell for that user, because .bashrc is sourced each time an interactive non-login shell starts — but it won't be visible to, say, a cron job or a systemd service, which don't go through .bashrc at all. /etc/environment is read by the PAM stack at login time system-wide, for every user, independent of shell — it's the right place for something that genuinely needs to be visible everywhere, including to non-interactive and non-shell processes.

The practical failure mode this question is probing for: someone sets an API key in ~/.bashrc, it works fine when they test it by hand, then a cron job or systemd service using the same variable fails because those don't source .bashrc — which is exactly the same category of "works interactively, fails under cron" surprise as the minimal-PATH issue in cron jobs.

bash(1) man page — INVOCATION section

It lists all running processes, filters for lines containing "myapp", and counts how many lines matched — the intent is almost always "how many instances of myapp are running." The bug is that the grep myapp command itself shows up as a process whose command line contains the string "myapp", so it matches its own search pattern and the count is off by one.

The fix is either to exclude grep's own line, or better, avoid the pattern-matching-your-own-command problem entirely with pgrep, which is purpose-built for this and doesn't have the self-match issue.

This is a favorite interview question specifically because it looks trivially correct and most candidates who haven't been bitten by it before will say the pipeline is fine — it tests whether you've actually debugged shell scripts in production, not whether you can read a pipeline.

pgrep(1) man page

An inode is a data structure that stores a file's metadata — permissions, owner, size, timestamps, and pointers to its actual data blocks — everything except the filename itself, which lives in the directory entry. Every filesystem allocates a fixed number of inodes when it's created, independent of the amount of disk space, so it's entirely possible to have plenty of free bytes but zero free inodes if there are enormous numbers of tiny files.

When inodes run out, you can't create a single new file even though df -h shows gigabytes free, because there's nowhere left to store that new file's metadata — the error is typically "No space left on device," which is misleading if you only check df -h and not df -i. This happens in practice on systems that generate huge numbers of small files, like session-cache directories, mail queues, or certain logging setups that write one file per event instead of appending to one file.

GNU coreutils — df

The core distinction is login shell versus non-login shell, and interactive versus non-interactive. A login shell (opening a fresh terminal that logs you in, or ssh user@host) sources .bash_profile (or .profile if .bash_profile doesn't exist); a non-login interactive shell (opening a new terminal tab inside an already-logged-in session) sources .bashrc instead. Because that split is confusing and easy to get wrong, the standard convention is to put actual settings — aliases, functions, PATH changes — in .bashrc, and have .bash_profile do nothing but source .bashrc, so you only maintain one file and both shell types end up with the same environment either way.

The scenario this shows up in: a PATH change that works in a fresh terminal but not over SSH (or vice versa) almost always traces back to it being defined in the wrong one of these two files for the shell type actually being used.

Bash manual — Bash Startup Files

Two separate steps, and mixing up their order is the most common mistake: first grow the logical volume itself, then grow the filesystem sitting on top of it to actually use that new space — resizing the LV alone doesn't make the filesystem bigger, it just gives the filesystem room to grow into. lvextend handles the first step and can be done live, without unmounting, as long as the volume group has free physical extents available. Which command handles the second step depends on the filesystem type: resize2fs for ext4, xfs_growfs for XFS — and XFS specifically can only grow, never shrink, which is worth knowing if someone asks the reverse question.

The trap: running lvextend and stopping there, then being confused that df -h still shows the old, smaller size — the filesystem resize is a separate, required second command.

lvextend man page

On Debian/Ubuntu, dpkg -S /path/to/file tells you which installed package owns that file; on RHEL/Fedora, the equivalent is rpm -qf /path/to/file. This matters in incident response and troubleshooting more than it sounds: if you find a suspicious or unexpectedly-modified binary, knowing which package it belongs to lets you compare it against a known-good version, check whether it's actually part of the OS at all (unowned files are a common indicator of something installed outside the package manager, worth extra scrutiny), and know what to reinstall to restore it.

dpkg(1) man page

strace traces the system calls a process makes to the kernel — every file open, read, write, network call, and so on — and prints them as they happen, along with their arguments and return values. It's the tool to reach for when a program fails or hangs with no useful error message of its own, because the system calls it makes are the ground truth of what it's actually trying to do, independent of whatever (possibly misleading) error the application chooses to print.

A concrete example: an application that fails with a vague "permission denied" or "file not found" error and no filename. Running it under strace and filtering for open/openat calls shows you exactly which file it tried to open and what error the kernel returned for that specific call, which usually points straight at the actual missing file or wrong permission — information the application's own logging never surfaced.

strace(1) man page

A soft limit is the currently enforced ceiling for a resource — like the number of open file descriptors or max processes — and any user can lower their own soft limit or raise it back up, as long as they don't exceed the hard limit. The hard limit is the absolute ceiling that only root can raise; an ordinary user can lower their own hard limit, but once lowered in a session they can't raise it back without root.

This two-tier design exists so a system administrator can set a sensible hard ceiling system-wide (via /etc/security/limits.conf) that ordinary users can't bypass, while still letting a user or application tune its own soft limit up to that ceiling if it legitimately needs more descriptors than the shell default provides — a database or web server hitting a low default soft limit is exactly the case where you'd raise it, without needing root, as long as it stays under the hard limit.

bash(1) man page — ulimit

bash myscript.sh explicitly invokes bash and hands it the script as an argument — it works regardless of the file's permissions or shebang line, because bash itself is doing the interpreting. ./myscript.sh asks the kernel to execute the file directly, which requires the file to have the execute permission bit set, and the kernel reads the first line for a shebang (#!/bin/bash) to know which interpreter to hand the rest of the file to; without a valid shebang, the kernel doesn't know what to do with it and execution fails.

The practical difference shows up when a script has no execute permission or the wrong shebang: ./myscript.sh fails with "permission denied" or "bad interpreter," while bash myscript.sh still runs it fine because permissions and shebang only matter for the direct-execution path, not when you explicitly name the interpreter.

execve(2) man page

A zombie is a process that has finished running but still has an entry in the process table because its parent hasn't yet called wait() to read its exit status — it's dead but not yet fully cleaned up. An orphan is the opposite situation: the child process is still running, but its parent died first, so the orphan gets automatically re-parented to init/systemd (PID 1), which is responsible for reaping it when it eventually exits.

Zombies are basically harmless individually — they use no CPU or memory beyond a process table slot — but a large accumulation of them signals a bug in the parent process that's failing to reap its children, and enough of them can exhaust the process table and block new processes from starting. You can't kill -9 a zombie because it's already dead; the only fix is making the parent call wait(), or if the parent itself is misbehaving, restarting the parent so its zombies get reaped.

wait(2) man page

Many long-running daemons — nginx, sshd, syslog implementations among them — treat SIGHUP not as its historical meaning ("the controlling terminal hung up") but as a convention meaning "reload your configuration without dropping what you're doing." nginx, for instance, uses SIGHUP to have its master process re-read nginx.conf and gracefully start new worker processes with the new config while letting old workers finish serving requests already in flight, rather than dropping active connections the way a full restart would.

This is purely a convention each daemon chooses to implement, not a kernel guarantee — sending SIGHUP to a process that hasn't specifically coded a handler for it just terminates it, using the signal's original default behavior. Always check a given daemon's own documentation for what it actually does with SIGHUP before relying on it, rather than assuming "reload config" is universal.

signal(7) man page

NFS is the traditional choice for Linux-to-Linux sharing: it's simple to set up, has decent performance for typical workloads, and is well understood, but it depends on the network being reliable — a flaky link can leave clients hung waiting on the mount rather than failing cleanly, and by default it doesn't encrypt traffic, which matters if the network isn't trusted. SMB/CIFS is the natural choice specifically when Windows clients are also involved, since it's native to Windows and NFS support there is comparatively clunky. For workloads that need to survive a node failing entirely rather than just a network hiccup, a distributed/clustered filesystem (GlusterFS, CephFS) replicates data across multiple nodes so the loss of one doesn't take the shared storage down with it — at the cost of meaningfully more operational complexity to run and reason about than a single NFS server.

The question to actually ask before picking one: what failure are you protecting against? A flaky network favors nothing (all of these degrade under network partition, though some more gracefully than others); a single point of failure on the storage node itself is what pushes you toward a distributed filesystem over plain NFS; and mixed Windows/Linux clients is what pushes you toward SMB regardless of the other trade-offs.

nfs(5) man page

Deleting a file with rm removes its directory entry, but the kernel only frees the underlying inode and data blocks once the link count and the open-file-descriptor count both reach zero. If some process still has the file open, the space stays allocated and invisible to du/ls, which is why df and du can disagree wildly after a big log file is "deleted" but the app that was writing to it is still running.

To find the culprit, use lsof (or fuser) to list open file descriptors that reference deleted files, then either restart the process cleanly (which releases the descriptor) or, if that's not an option, truncate the file descriptor directly through /proc.

The trap is reaching for rm -rf again or checking du a second time and concluding the problem "fixed itself" — it won't, until the holding process closes or is restarted. This is one of the most common causes of a disk that's "full" according to df but empty according to du -sh /*.

lsof man page

at schedules a genuinely one-off job ("run this once at 2:05pm") but the atd daemon has to be running and the job is lost if the box reboots before it fires and you didn't persist it. cron is built for recurring schedules; you can fake a one-off with it by having the job remove its own crontab entry after running, but that's a workaround, not what cron is for. A systemd timer unit paired with a oneshot service is the more robust modern answer: it integrates with the same dependency and logging system as the rest of systemd (journalctl shows its output natively), it can be configured with Persistent=true so a missed run (because the machine was off) fires as soon as the system is back up, and systemctl list-timers gives you a clean audit trail of what's scheduled and when it last ran — none of which cron or at give you for free.

The trade-off to name explicitly: systemd timers are more verbose to set up (two unit files instead of one crontab line) and not every environment has systemd, so on older or minimal distros cron/at may be the only option. For "one job in five minutes and I don't care about surviving a crash," at is legitimately the simplest correct answer — the point of this question isn't that one tool is always right, it's whether you can name the actual trade-offs instead of just reciting "systemd timers are better."

systemd.timer man page

Work outward from the process, layer by layer, so each step rules something out rather than guessing at the whole stack at once. First confirm the process is actually listening on the interface you expect, not just on loopback — ss -tlnp (or the older netstat -tlnp) shows the bind address; 127.0.0.1:8080 means it will never accept an external connection no matter what the firewall does, while 0.0.0.0:8080 means it's listening on all interfaces. If it's bound correctly, check the host firewall next — iptables -L or, on distros using it, firewall-cmd --list-all or ufw status — since a default-deny rule on the port is the single most common cause of "works locally, not remotely." If the firewall looks fine, test from the remote host with curl or telnet host port to see whether the TCP handshake even completes; if it hangs rather than refusing, that points at something upstream — a cloud security group, a network ACL, or a router — rather than the host itself. tcpdump on the target host during a remote connection attempt is the tiebreaker: if the SYN packet never arrives, the problem is upstream of the box; if it arrives and the host doesn't respond, the problem is local (app or firewall).

The trap is assuming "works with curl localhost" already proves the app is fine — it only proves the app works when there's no network or firewall layer between client and server at all, which is exactly the part that's broken.

ss(8) man page

Start with du scoped sensibly rather than scanning the whole tree at once — du -x -h --max-depth=1 / (the -x stops it wandering into other mounted filesystems and double-counting) tells you which top-level directory is the offender, then repeat one level deeper into that directory until you find the actual large files or directories. Common repeat offenders on a long-running box: /var/log (unrotated or verbose logs), old kernel packages left behind after upgrades, Docker's image/layer cache under /var/lib/docker, and core dumps under /var/crash or wherever core_pattern points.

Once you know the cause, the fix without downtime depends on what it is. Logs: rotate and compress them in place with logrotate, or if a single huge log file needs to shrink immediately without restarting the process writing to it, truncate the contents while keeping the file descriptor open — truncate -s 0 /var/log/app.log — rather than deleting the file, which as covered earlier would just leave the space held by the running process instead of freeing it. Old kernels/packages: apt autoremove or dnf autoremove reclaims real space safely. Docker: docker system prune clears dangling images and stopped containers. If nothing can be safely removed and the partition is genuinely undersized, the only real fix is extending the underlying volume — if it's on LVM, lvextend plus resize2fs/xfs_growfs can grow the filesystem live, without unmounting, as long as there's free space in the volume group.

The trap is reaching for rm on a log file a process still has open — that "frees" space in df output that du will still show as gone but which df won't actually release until the process is restarted, which is exactly the scenario this question is testing whether you remember.

resize2fs man page

Disable password authentication and require key-based login (PasswordAuthentication no) — this alone stops the overwhelming majority of automated brute-force attempts, since guessing a private key is computationally infeasible in a way guessing a password isn't. Disable direct root login (PermitRootLogin no) so an attacker who does get in has to compromise a named account first and then escalate, which is both harder and leaves an audit trail, rather than landing directly on the account with unlimited privileges. Move SSH off port 22 only helps against the laziest scanners and shouldn't be relied on as real security — it's worth mentioning as a minor noise-reduction measure, not a control, because pretending it is one is a common weak answer. More substantial: restrict which users or groups can connect at all with AllowUsers/AllowGroups, and put fail2ban (or an equivalent) in front of the service to automatically block IPs after repeated failed attempts, which catches the credential-stuffing pattern that a single failed-login threshold on its own doesn't stop.

The trap in this question is a candidate who lists port-changing as their first or only answer — it signals they're optimizing for "looks like a security answer" rather than understanding what actually reduces attack surface versus what just reduces log noise. Also worth flagging: always test a new SSH config in a second, still-open session before closing the one you're editing in — a syntax error in sshd_config that locks out the only open session on a remote box with no console access is a classic self-inflicted outage.

OpenSSH sshd_config man page

A few choices here aren't stylistic, they're correctness: set -euo pipefail makes the script exit on the first unhandled error (-e), treat unset variables as errors instead of silently expanding to empty strings (-u), and propagate failure through a pipeline instead of only checking the exit code of the last command in it (pipefail) — without that last flag, a failure in find would go unnoticed as long as the while loop itself succeeded. Quoting "$LOG_DIR" and "$file" isn't optional either; an unquoted variable that happens to contain a space or glob character gets word-split and glob-expanded by the shell, which is exactly the kind of bug that only shows up once, in production, on a filename nobody tested with. Using find -print0 piped to read -d '' (null-delimited) instead of a plain newline-separated loop is the same defensive habit applied to filenames that might contain spaces or even newlines — a plain for file in $(find ...) breaks on those.

The trap in this question isn't the find syntax, it's whether a candidate's script quietly breaks the moment a filename has a space in it, which is common enough in real log directories to matter.

Bash manual — Set Builtin

Plain partitions are the simplest option: fixed size at creation time, no extra abstraction layer, easiest to reason about and recover — the right default when the storage layout is genuinely static and you don't expect to resize anything later. LVM (Logical Volume Manager) adds a layer of logical volumes on top of physical disks, which buys you the ability to grow (and with some filesystems, shrink) a volume live without unmounting it, take snapshots for consistent backups, and span a single logical volume across multiple physical disks — the cost is a small amount of added complexity and one more layer to understand when something goes wrong at 2am. RAID is a different axis entirely: it's about redundancy and/or performance across multiple physical disks, not flexible resizing — RAID 1 mirrors for redundancy, RAID 5/6 trade some capacity for redundancy plus better read performance, RAID 0 stripes for pure performance with zero redundancy (one disk failure loses everything).

They're not mutually exclusive — a very common production layout is RAID for the redundancy guarantee, with LVM on top of the RAID array for flexible resizing, and ordinary partitions or filesystems on top of that. The answer that shows real experience isn't picking one and defending it, it's naming which axis each tool actually solves: partitions for simplicity, LVM for flexibility, RAID for redundancy/performance, and combining them when you need more than one of those properties at once.

Red Hat — LVM Administration

Ext4 supports offline shrinking because its on-disk layout allows the filesystem's metadata and data blocks to be relocated and the filesystem's boundary redrawn smaller — you unmount it, run resize2fs to shrink the filesystem first, then lvreduce to shrink the logical volume to match (shrink the filesystem before the volume, or you risk truncating live data). XFS's on-disk format was designed around growing, not shrinking, and the tooling simply never implemented a shrink path — xfs_growfs only grows, there is no xfs_shrink.

If an XFS volume genuinely needs to end up smaller, the practical answer isn't a shrink operation at all: create a new, smaller logical volume, format it XFS, copy the data across (rsync -a is the standard tool for this since it preserves permissions and can resume), then swap the mount point and remove the old volume. That's more disruptive than an ext4 shrink, which is exactly why it's worth choosing ext4 over XFS up front for any volume you might realistically need to shrink later, and it's the kind of trade-off worth naming explicitly rather than assuming XFS and ext4 are interchangeable.

XFS FAQ — shrinking

This is almost always a file-descriptor leak: the process opens files, sockets, or pipes over time and doesn't close them, so the count climbs toward a per-process limit (ulimit -n) until new opens start failing — the intermittent, slow-building nature of the symptom (versus an immediate crash) is itself the clue that it's a leak and not a one-off misconfiguration. Confirm it first: lsof -p <PID> | wc -l right after a restart versus the same command an hour later should show the count climbing steadily rather than staying flat, which distinguishes a genuine leak from a workload that legitimately needs more descriptors than the current limit allows.

If it's a leak, lsof -p <PID> shows what the growing descriptors actually are — a huge number of identical entries (the same socket type, or the same log file path opened repeatedly) points at exactly which code path is failing to close what it opens, which is where you'd go looking in the application's code or its library dependencies. If it turns out the workload is legitimate and just needs more file descriptors than the OS default allows — a high-connection-count server is a normal case for this — the fix is raising the limit rather than chasing a leak that doesn't exist: ulimit -n for the current shell, or a persistent nofile limit in /etc/security/limits.conf or the systemd unit's LimitNOFILE= for a service.

The trap is jumping straight to raising the limit without confirming whether the count is actually climbing — that masks a genuine leak for longer instead of fixing it, and the problem returns at a higher, harder-to-reach ceiling.

limits.conf(5) man page

Start by narrowing the time window before touching anything, since guessing at causes without a timeframe means checking everything. journalctl --since "<time>" scoped to roughly when the problem started (from a ticket, a monitoring alert, or the user's report) surfaces service restarts, kernel messages, and errors around the actual window, cutting out months of irrelevant log noise. In parallel, check what actually changed on the filesystem: package manager logs (/var/log/apt/history.log on Debian/Ubuntu, /var/log/dnf.log or rpm -qa --last on RHEL/Fedora) show what was installed, upgraded, or removed and when, which is very often the actual root cause of "it used to work" — an unattended security update silently bumping a library or runtime version is one of the most common real-world causes of this exact complaint. If package history doesn't explain it, check for recent config file edits under /etc by modification time, and check crontab -l / systemctl list-timers for anything scheduled that might have run once and changed state.

The trap in this question is a candidate who jumps straight to "check the logs" without first narrowing a time window — on a box that's run for months, an unscoped log search buries the actual signal in noise. What this question is really testing is whether you instinctively narrow scope first (when did it last work, what changed since) before diving into any single tool.

journalctl man page

Older container tooling and some applications read memory limits from /proc/meminfo, which reports the host's total memory, not the cgroup limit imposed on the container — a container capped at 512MB by its cgroup can still read "64GB available" from /proc/meminfo if the JVM, Node, or another runtime inside it makes memory-sizing decisions based on that file instead of the actual cgroup limit. That mismatch is exactly what causes an application to size its heap or cache far larger than its real ceiling, run fine for a while, then get killed the moment it actually tries to use memory the cgroup won't grant it — which looks "unpredictable" from outside but is entirely deterministic once you know what triggered it.

The fix is making the application read its actual limit from the cgroup interface instead of /proc/meminfo — modern JVMs (Java 10+) and recent versions of most major runtimes do this automatically once cgroup-awareness is enabled; older runtimes need an explicit flag or an environment variable pointing at the cgroup limit. The cgroup limit itself is readable directly and is the source of truth regardless of what /proc/meminfo says.

The trap is debugging this as a "the container is misconfigured" problem and bumping the memory limit repeatedly — that just delays the same OOM kill, because the actual bug is the application sizing itself off the wrong number, not the limit being too low.

cgroups(7) man page

Create a dedicated service account with a restricted or no-op login shell — /usr/sbin/nologin — so the account can authenticate for the specific purpose it needs (SSH key auth for SCP/rsync, for instance) but can't be used to open an interactive shell session even if the key were compromised. Pair that with SSH key restrictions in authorized_keys rather than relying on the account alone: the command= option forces a specific command to run regardless of what the connecting client actually requests, and no-pty, no-agent-forwarding, and no-port-forwarding close off the other things a compromised key could otherwise be used for beyond the one intended command.

Combine that with filesystem permissions that make the intent enforceable rather than just documented: the service account's own group only gets write access to the specific deploy directory, and the underlying files it deploys should ideally be owned by a different, unrelated account so a compromised deploy key can overwrite the next deployment but can't tamper with what's currently serving traffic. The layered thinking is the actual answer here — no single control (the nologin shell, the forced command, the directory permissions) is sufficient by itself, and a candidate who names only one of them hasn't fully solved the actual problem.

OpenSSH authorized_keys documentation