Linux Interview Questions

The shell is the program that reads and runs your commands (bash, zsh, dash), the terminal is the window or device that shows text and sends your keystrokes, and the console is the machine's primary text interface, either physical or virtual. They are separate layers that people mix up.

When you open a terminal app, it starts a shell inside it. Over SSH, the terminal is on your laptop and the shell runs on the server. You can swap one without the other, for example running zsh inside the same terminal.

A common trap: $SHELL shows the login shell, not necessarily the shell you are in. If you type zsh inside bash, $SHELL still says bash.

chown changes the owner (and optionally the group) of a file, and chgrp changes only the group. Only root can give a file to another user, but the owner can change the group to any group they belong to.

-R walks the whole tree, so a wrong path or an unset variable can change ownership of a huge area. The classic accident is chown -R user .*, because .* also matches .., the parent directory, and the command climbs out of the folder you meant. Check the path first with ls, and be careful with symlinks: by default -R does not follow them, which is the safe behaviour.

Use the manual pages first (man), then the command's own --help, then info and the files under /usr/share/doc. All of these work offline.

Man pages are split into numbered sections: 1 is user commands, 5 is file formats, 8 is admin commands. The same name can exist in several sections, which is why passwd and man 5 passwd are different pages. A trap: man cd often shows the big bash manual, because cd is a builtin. Use help cd instead.

Linux identifies users by a number, the UID, and groups by a number, the GID. Names are only labels. UID 0 is root, the account the kernel treats as all-powerful, and the check is on the number, not on the name "root".

Every user has one primary group and can have extra (supplementary) groups. Permission checks use the process's UID, its primary GID and its supplementary groups.

The last command is a quick security check. Only root should have UID 0. A second account with UID 0, even with an innocent name, is effectively another root and is a classic sign of a backdoor.

curl is a flexible tool for talking to URLs and APIs (any HTTP method, headers, many protocols), while wget is built mainly for downloading files, including resuming and mirroring whole sites. For API work, curl is the usual choice.

The big trap is that plain curl exits with 0 even on an HTTP 404 or 500, because the transfer itself worked. Add -f (--fail) in scripts so an HTTP error becomes a non-zero exit code. Also, curl ... | bash runs whatever the server sends. Download first, read it, then run it.

A foreground job owns your terminal, and a background job runs without taking the prompt. The shell tracks both in a job list for that session.

Ctrl-C sends SIGINT (interrupt) and Ctrl-Z sends SIGTSTP (pause), so Ctrl-Z does not end the program, it only stops it until you resume it. Job numbers like %1 belong to one shell session. A background job usually dies when that session ends, so use nohup, disown or tmux for anything that must outlive your login.

They differ mainly in package format and tools, release and support style, and default security and firewall tooling. Skills transfer well, but commands and file paths change.

Rocky and AlmaLinux are rebuilds of RHEL, and Ubuntu has long-term support (LTS) releases that are common on servers. The trap is writing scripts that assume apt or /var/log/syslog. Detect the system with /etc/os-release or write for the family you actually run.

Use lscpu, lsmem, lspci, lsusb and dmidecode, which read what the kernel and firmware report. Together they answer "what is this machine?".

On a virtual machine, dmidecode shows the hypervisor's virtual hardware, not real chips, which is useful to know when a cloud instance behaves differently from a bare-metal server. lshw gives a combined tree if it is installed.

Never cat a multi-gigabyte file. Use a pager (less) or look at the edges with head and tail, and search with grep instead of opening it in an editor.

less does not load the whole file into memory, but a normal editor such as vim may. If a binary file scrambles your terminal after cat, run reset to restore it. For wide lines, less -S stops wrapping so you can scroll sideways.

Use diff to see what differs, and patch to apply a saved difference. The unified format (-u) is the standard because it shows context and is readable in code review.

The last line uses process substitution to compare a file on two servers without copying anything, which is very handy for spotting configuration drift. Use cmp when you only need to know whether two binary files are identical, and diff -q for a quick yes or no on directories.

rm removes a name from a directory, and there is no recycle bin, so the data is gone. rmdir only removes empty directories, and rm -rf removes a whole tree without asking.

Habits that prevent disasters: run ls with the same pattern first, quote variables, and never run rm -rf "$DIR/" in a script unless you are sure $DIR is set. If $DIR is empty, that becomes rm -rf /. GNU rm refuses to remove / itself (--preserve-root is on by default), but that does not protect /* or your home directory.

/tmp is for short-lived files that may be deleted at reboot (and is often RAM-backed tmpfs), while /var/tmp is for temporary files that must survive a reboot. Both are world-writable with the sticky bit.

Which one is cleaned and when depends on the distribution. On systemd systems, the rules in /usr/lib/tmpfiles.d/tmp.conf decide when old files are removed, and /var/tmp is usually kept much longer (around 30 days).

Do not keep anything important in /tmp. If /tmp is a tmpfs, a program that fills it also uses memory, which can push the machine into swap.

A system call is the controlled way a normal program asks the kernel to do something it is not allowed to do itself, such as open a file, send network data or create a process. User space is where applications run with limited rights, and kernel space is where the kernel runs with full access to hardware and memory.

A program cannot touch the disk or network directly. It calls a library function (like open() in the C library), which triggers a system call, the CPU switches into kernel mode, the kernel does the work and returns the result.

This is why strace is so useful: whatever an application claims, its system calls show what it really asked the kernel to do. System calls are also a performance cost, since each one is a switch into the kernel, which is why programs batch reads and writes.

A target is a named group of units that represents a system state, and it replaces the old SysV runlevels. For example, multi-user.target is the normal text-mode server state and graphical.target adds a desktop.

To boot into rescue mode once, add systemd.unit=rescue.target to the kernel line in GRUB. Most servers use multi-user.target, because they do not need a graphical interface and it saves memory and attack surface.

Use lsblk for the device tree, blkid for UUIDs and filesystem types, and findmnt or df for what is mounted and how full it is. Each answers a different question.

Device names like /dev/sda and /dev/sdb can change between boots, depending on detection order, so use the UUID (or a label) in /etc/fstab. A common mix-up: lsblk shows devices that exist, while df only shows filesystems that are mounted, so an unmounted or unformatted disk will appear in the first but not the second.

Try to open a connection with nc, curl or Bash's /dev/tcp, and read the result: success, "connection refused" or a timeout. The three outcomes mean different things.

• Succeeded: something is listening and the path is open.
• Connection refused: the packet arrived, but nothing is listening on that port (or a firewall actively rejects).
• Timeout: packets are being dropped, usually by a firewall, security group or routing problem.

Remember to test from the right place: the machine that will really make the connection, not just your laptop. UDP is harder to test, because no reply is not proof of failure (nc -vzu is only a rough hint).

Use ss to list listening sockets with the process that owns them, then stop or reconfigure that process. ss is the modern replacement for netstat.

You need root to see process names for other users. If your app fails with "address already in use", usually another instance is still running (an old process or a duplicate service), or the port is in the TIME_WAIT state after a restart and the app did not set SO_REUSEADDR. Find the PID, check what it is (ps -fp PID), and stop it properly with systemctl stop rather than kill -9 where possible.

For a listening socket, Recv-Q and Send-Q in ss -ltn mean the current accept queue and its maximum size, which is a different meaning from established connections.

MBR is the old scheme, limited to 2 TiB disks and four primary partitions, while GPT is the modern scheme that supports huge disks and many partitions and keeps a backup copy of its header. Use GPT for new disks.

UEFI systems need an EFI System Partition (ESP), a small FAT filesystem that holds boot loaders. A trap: creating a new partition table wipes the old one, so double-check the device name with lsblk before running parted, fdisk or mklabel.

ext4 is the safe general-purpose default, XFS is excellent for large files and heavy parallel I/O, btrfs and ZFS add snapshots and checksums, and tmpfs lives in RAM. Choose by workload and by the features you need.

• ext4: mature, simple, can grow and shrink (offline). Good default for the root filesystem.
• XFS: scales well to very large filesystems and many parallel writers, can grow online but never shrink. Default on RHEL.
• btrfs: snapshots, checksums, built-in compression, subvolumes. More moving parts.
• ZFS: very strong data integrity, snapshots and replication, but it is not in the mainline kernel and has its own licence and memory needs.
• tmpfs: RAM-backed, for /run and /tmp.
• vfat or exFAT: removable media and the EFI boot partition.
• overlayfs: layered filesystem used by container images.

The decision usually comes down to "will I need to shrink it?" (ext4), "huge files or databases with many threads?" (XFS), and "do I need snapshots and checksums built in?" (btrfs or ZFS). When no special need exists, the default of your distribution is the lowest-risk choice.

Both give cryptographically secure random bytes from the kernel. On modern kernels (5.6 and later), /dev/random only blocks until the random generator is initialised at boot, and after that it behaves the same as /dev/urandom.

Older advice said "use /dev/random for keys because /dev/urandom may run out of entropy". That is outdated: once the kernel's generator is seeded, it does not "run out". The real risk is very early boot, especially on virtual machines, when the generator has not yet collected enough randomness.

In code, call getrandom() (or your language's secure random API), which blocks only until the generator is ready, instead of reading the device files. For ordinary key and token generation, /dev/urandom is correct.

chroot changes a process's idea of the root directory, so it can only see files below a chosen folder. It is useful for rescue work and builds, but it is not a strong security boundary.

A process with root rights inside a chroot can often escape it, because chroot does not restrict system calls, devices or privileges. It also does not isolate processes, network or users. Containers use namespaces, cgroups, capability limits and seccomp to provide real isolation. So use chroot for repairs and package builds, and use containers or VMs (or systemd sandboxing) when isolation matters.

ping sends ICMP echo requests and tells you whether a host answers and how long it takes. traceroute shows the routers along the path by sending packets with increasing TTL values. Both use ICMP or UDP, not your application's TCP traffic.

ping failing does not mean the server is down. Many firewalls and cloud security groups block ICMP while allowing TCP, so a web server can be healthy yet unpingable. The opposite also happens: ping works but the application port is blocked. So test the real port (nc -vz host 443 or curl) to prove the service is reachable. In traceroute, lines of * * * often just mean a router ignores these probes, not that traffic stops there.

remove uninstalls a package but keeps its configuration files, purge removes the package and its configuration, and autoremove removes dependencies that nothing needs any more.

Keeping the config on remove is useful if you reinstall later, but it can leave old settings behind that surprise you, so purge when you truly want a clean slate. On RHEL, dnf remove removes the package, and dnf autoremove removes unneeded dependencies (config files that were edited are left as .rpmsave files). Read the list before you press yes on autoremove, and use apt-mark hold for packages that must not change.

Private ranges are addresses reserved for internal networks that are not routed on the public internet, and CIDR notation like 10.0.0.0/24 says how many leading bits identify the network. The number of addresses is 2^(32 - prefix).

A /24 has 2^8 = 256 addresses, of which 254 are usable hosts (the first is the network address, the last is broadcast). A /16 has 65,536, and a /28 has 16.

Also know 169.254.0.0/16 (link-local, shows up when DHCP fails) and 100.64.0.0/10 (carrier-grade NAT). A real-world trap is overlapping ranges: two networks that use 10.0.0.0/16 cannot be joined by a VPN or peering without renumbering.

Single quotes keep everything literal, double quotes still expand $variables, $(commands) and backticks, and a backslash escapes one character. Quoting is how you stop the shell from splitting and globbing your text.

The most useful rule is "always quote variable expansions": "$var". Unquoted, the value is split on whitespace and any * in it is expanded to file names. When you send a command through SSH, ssh host "echo $HOME" expands $HOME on your machine first. Use single quotes (ssh host 'echo $HOME') if the remote side should expand it.

They are positional parameters: $1, $2 are the arguments, $# is how many there are, and "$@" expands to each argument as a separate, intact word. "$@" is the form that forwards arguments correctly.

• "$@" keeps hello world as one argument.
• "$*" joins everything into one string (using the first character of IFS).
• Unquoted $@ or $* is re-split on spaces, so hello world becomes two arguments.

Use shift to drop $1 and move the rest down, and $0 for the script name. A wrapper script that calls another program should almost always end with exec "$@".

$(cmd) captures a command's output as text and is the modern form that nests cleanly, while backticks do the same but are harder to nest and read. <(cmd) is different: it makes a command's output look like a file, which is useful for tools that expect file names.

Command substitution strips trailing newlines, which is usually what you want. Process substitution is a bash, zsh and ksh feature and does not exist in plain POSIX sh, so a script with #!/bin/sh may fail on it. >(cmd) is the reverse: it gives a file name that feeds a command.

They let you set defaults, require values and trim strings without calling sed or cut, so scripts stay fast and simple.

Know the difference between :- and -: with the colon, an empty value also triggers the default, without it only an unset variable does. With set -u on, reading an unset variable is an error, so use ${var:-} when "unset" is acceptable.

The redirection > is done by your own shell, before sudo runs, so the file is opened with your privileges, not root's. Only echo runs as root, and by then it is too late.

tee reads standard input and writes it to the file, and since tee itself runs under sudo, it has the right to write. For editing a root-owned file interactively, sudoedit (or sudo -e) is safer than sudo vim.

Bash checks in order: aliases, shell keywords, functions, builtins, then files found in PATH (using a cache of earlier lookups). The first match wins, which explains many "why is it running the wrong thing" bugs.

which is not a reliable answer, because it is an external program that does not know about aliases, functions or builtins. After you move or install a binary and the shell still runs the old path, the cached location is the cause, and hash -r fixes it. Scripts do not expand aliases by default, so an alias that works in your terminal will not work inside a script.

A script runs in a child process, and a child cannot change its parent's directory or environment. When it exits, those changes disappear. source script.sh (or . script.sh) runs the commands inside your current shell instead, so the changes stay.

This is why nvm, Python virtualenv activate and similar tools must be sourced, not executed. A handy trick inside scripts is the subshell ( cd dir && command ), which runs a command elsewhere without changing the script's own directory. The reverse also matters: sourcing an untrusted file runs its code with your shell's full privileges.

A here-document (<<) feeds several lines of text to a command's standard input, written right in the script. It is the clean way to create config files or send multi-line input without temporary files.

Quoting the marker (<<'EOF') stops variable and command expansion, which is usually what you want for config files. Unquoted <<EOF expands $variables, which is useful for templates but surprising otherwise. <<-EOF strips leading tab characters (not spaces), so you can indent the block. Two traps: the closing marker must be alone on its line, and indenting it with spaces breaks it.

Use while IFS= read -r line; do ...; done < file. The for ... in $(cat file) form splits on every space, not on lines, and also expands wildcards in the data.

• IFS= stops leading and trailing spaces being trimmed.
• -r stops backslashes being treated as escapes.
• || [ -n "$line" ] still processes a last line that has no trailing newline.

Two classic traps. First, cat file | while read ... runs the loop in a subshell, so variables set inside are lost afterwards. Use done < file or done < <(command) instead. Second, if the loop body runs ssh, ssh reads the rest of the file from standard input and the loop ends after one iteration. Use ssh -n or < /dev/null.

set -e exits on a failing simple command, but it is deliberately switched off in several situations, so it gives a false sense of safety. Do not rely on it alone.

Cases where a failure does not stop the script:

• A command in an if or while condition, or on the left of && or ||.
• Any command in a pipeline except the last, unless set -o pipefail is on.
• A command preceded by !.
• local x=$(failing_cmd), because local returns success and hides the error. Declare first, assign second.
• (( i++ )) when the value is 0, which returns status 1 and can kill the script unexpectedly.

Good scripts combine strict mode with explicit checks on the important steps and a trap ... ERR or EXIT handler for cleanup and clear error messages.

Check the syntax without running it, trace the commands as they execute, and lint it with ShellCheck. Tracing shows exactly what the shell expanded and ran.

Inside a script you can trace one region with set -x ... set +x. A trap 'echo "failed at line $LINENO" >&2' ERR pinpoints where an error happened. Be careful: set -x prints variable values, so secrets such as tokens and passwords can leak into CI logs. ShellCheck catches most real-world bugs, such as unquoted variables, before the script ever runs, so it is worth running in CI.

POSIX sh is the small standard shell language, and Bash is a larger shell with extra features called bashisms. On Debian and Ubuntu, /bin/sh is dash, a fast minimal shell without those extras, so a Bash-only script run as sh fails.

Common bashisms: [[ ... ]], arrays, == inside [ ], <<<, <(...), {1..5}, source (use .), echo -e, $'...', function name {}.

Use #!/usr/bin/env bash when you want Bash features, and #!/bin/sh only if the script is truly POSIX. Containers based on Alpine have BusyBox ash and often no Bash at all, which is a frequent cause of "works on my laptop, fails in the container".

jq is a filter language for JSON: you describe the path or condition and it prints the result. Never parse JSON with grep or sed, because it breaks on whitespace, nesting and escaping.

Put the jq program in single quotes so the shell does not touch $ and .. Use --arg instead of building the filter by string concatenation, which avoids injection and quoting bugs. -e makes jq usable in if conditions and CI health checks.

Wrap the command in flock, which takes an exclusive lock on a file. If the previous run is still going, the new run exits (with -n) or waits. The lock is released by the kernel when the process ends, even if it crashes.

This beats a hand-made PID file because a PID file left behind after a crash blocks the job forever, and checking it is racy. Combine flock with timeout 50m command so a hung job cannot hold the lock forever. Do not put lock files on NFS unless you know the lock semantics, since flock behaviour over network filesystems varies.

xargs -P N runs up to N commands at the same time, and -n 1 or -I{} controls how items are passed. It is the simplest way to parallelise a loop with no extra tools.

Things to know: output from parallel jobs can interleave, so write each job's output to its own file or use GNU parallel which groups it. xargs exits with 123 if any command failed, so check that in scripts. Use -r (GNU) so nothing runs when the input is empty. Choose -P based on the bottleneck: CPU-bound jobs near the core count, network or disk jobs lower to avoid overload.

They are three regex dialects. Basic (BRE) is the default in grep and sed, Extended (ERE) is used with grep -E and awk, and Perl-compatible (PCRE) is used with grep -P and adds features like \d and lookahead.

The common trap is using \d in grep -E and wondering why it matches nothing: \d exists only in PCRE. Use [0-9] or [[:digit:]] for portability. grep -P is not available on every system (macOS grep lacks it). Also remember that regex . means "any character" and * means "repeat the previous item", unlike shell globs.

Locale settings control character encoding and sort order, and Windows line endings (CRLF) leave a hidden carriage return at the end of each line. Both produce bugs that look like random failures.

The classic error is bad interpreter: /bin/bash^M, caused by a script saved with CRLF. Sort order also depends on locale: in en_US.UTF-8 upper and lower case are mixed, while in C all capitals come first. That mismatch can make comm or join give wrong results when inputs were sorted under different locales. Set LC_ALL=C for scripts that need predictable, fast text processing.

Many programs can start a shell or run commands, so permission to run them as root is effectively full root access. These shell escapes are well documented for common tools.

The same applies to editors, pagers, interpreters (python, perl), tar, zip, env, and any wildcard in a sudoers argument. Mitigations: allow sudoedit (which edits a copy as your own user) instead of sudo vim, allow only exact commands with exact arguments, use the NOEXEC tag for programs that must not spawn children, and prefer small wrapper scripts you control. The site GTFOBins lists which binaries can be abused this way, and is useful when reviewing a sudoers file.

PAM (Pluggable Authentication Modules) is the layer that programs like sshd, login, su and sudo call to decide who may log in and under what rules. Each program has a config file in /etc/pam.d/ listing modules in order.

A PAM rule has four kinds of stack: auth (prove identity), account (is this account allowed now), password (changing passwords) and session (setup and cleanup, such as limits). Each line has a control flag: required, requisite, sufficient or optional.

Useful modules: pam_faillock (lock after failed tries), pam_pwquality (password rules), pam_limits (applies limits.conf), pam_access (restrict by user or host). The danger is that a typo can lock everyone out, including root. Always keep a root session open while testing PAM changes, and on RHEL use authselect instead of editing generated files by hand.

Use chage for aging and expiry, passwd -l or usermod -L to lock the password, and usermod -s /usr/sbin/nologin to remove shell access. These do different things, so choose the one that matches your goal.

The trap: locking the password usually does not stop SSH key login, because key authentication does not use the password. To truly disable a departing employee, expire the account (chage -E 0), remove their SSH keys, and kill their sessions (loginctl terminate-user alice).

System accounts are non-human accounts for running services, with low UIDs, usually no home directory and no login shell. Regular users are people, with UIDs starting at 1000 on most modern distributions (500 on older RHEL).

The ranges come from /etc/login.defs. Create a service account with useradd -r:

Giving each service its own account limits damage: if the web server is exploited, the attacker has only that account's file access, not the database's or the admin's. Do not share one account across services, and never run network-facing daemons as root. In systemd you can go further with User= or DynamicUser=yes, which creates a temporary account for the life of the service.

The Linux audit system records kernel-level events according to rules you define. You add a watch on the file, then search the audit log by key.

To make rules survive reboots, put them in /etc/audit/rules.d/*.rules and load them with augenrules --load. The log records auid (the original login user), which stays the same even after sudo or su, so you can see which person was behind a root action. Trade-offs: logging every execve produces a lot of data, so scope rules carefully, forward logs to a central server, and consider locking rules with -e 2 so an attacker cannot change them without a reboot.

Capabilities split root's power into small separate privileges, so a program can have only the one it needs. CAP_NET_BIND_SERVICE lets a process bind to ports below 1024 without being full root.

In systemd you grant them per service:

Know the dangers. CAP_SYS_ADMIN is so broad it is called "the new root". Never setcap an interpreter such as python, because then every script run by it gets the power. File capabilities can be lost when files are copied without extended attributes. Containers drop most capabilities by default, which is part of their hardening.

SSH remembers each server's host key fingerprint in ~/.ssh/known_hosts the first time you connect (trust on first use). If the key later differs, SSH stops you because it cannot tell a legitimate change from a man-in-the-middle attack.

Harmless causes: the server was rebuilt, the IP address was reused by a different machine, or a load balancer now sends you to another backend. Dangerous cause: someone is intercepting the connection.

The right reaction is to verify the new fingerprint through another channel (cloud console, the server admin) before removing the old line. Do not make StrictHostKeyChecking=no your habit, because it turns the warning off for real attacks too. At scale, use SSH host certificates so clients trust a CA instead of each individual key.

Define hosts, users, keys and jump hosts once in ~/.ssh/config, and then ssh prod-db is enough. The file is read top to bottom and the first value found for each option wins, so put specific hosts before Host *.

IdentitiesOnly yes stops SSH trying every key in your agent, which can hit the server's MaxAuthTries limit and cause "Too many authentication failures". ControlMaster reuses one connection for many sessions, so repeated ssh and scp calls are much faster. ProxyJump replaces older ProxyCommand tricks for reaching private hosts through a bastion.

Identify the disk, create a partition table and partition, make a filesystem, mount it, and add a UUID-based line to /etc/fstab. Test the fstab line before rebooting.

Notes that show experience: confirm the device name with lsblk and size first, because formatting the wrong disk destroys data. Use the UUID, not /dev/sdb1, because names can change. nofail stops a missing non-essential disk from blocking boot. For something that will need resizing later, use LVM on the disk instead of a plain partition.

The journal is a log where the filesystem writes what it is about to change before changing it. After a crash, the filesystem replays or discards the log, so its structure is consistent again within seconds, without scanning the whole disk.

Without a journal, a power cut during a write could leave a half-updated directory or lost blocks, and a full fsck would have to read every inode, which takes a very long time on large disks.

On ext4, the default mode is data=ordered: only metadata (names, sizes, block maps) goes through the journal, but file data is written before the metadata that points to it. You will see recovering journal in the kernel log after an unclean shutdown.

Important limit: the journal protects the filesystem's consistency, not your application's data. An application that must not lose data has to call fsync() to force it to disk, which is why databases do this carefully. Never run fsck on a mounted filesystem.

Ext filesystems reserve a share of the blocks (5% by default) for root and system daemons. df hides that reserved space from the "Avail" column, so normal users hit "no space left" while root can still write.

Why the reserve exists: it keeps a system usable when users fill the disk, so root can log in, daemons can write logs and you can still repair things, and it also reduces fragmentation. On a 4 TB data disk 5% is 200 GB, so lowering it to 1% for a data-only filesystem is reasonable. Do not set it to 0 on the root filesystem. XFS reserves only a very small amount, so this behaviour is mainly seen on ext4.

This is also one more cause to check when du and df disagree, along with deleted-but-open files and inode exhaustion.

A snapshot is a copy-on-write view of a logical volume frozen at a moment in time. It stores only the blocks that change afterwards, so it is created almost instantly and takes little space at first.

Points to know:

• Size matters. The 5 GB is space for changes. If the origin changes more than that while the snapshot exists, the snapshot becomes invalid.
• Performance cost. Writes to the origin are slower while a classic snapshot exists, so remove it when finished.
• Consistency. A snapshot is crash-consistent, like a power cut. For databases, flush or lock tables first (or use the database's own backup tool), or use fsfreeze.
• Rollback. lvconvert --merge reverts the origin to the snapshot, which is useful before risky upgrades.
• A snapshot on the same disk is not a backup, because a disk failure destroys both.

LUKS (managed by cryptsetup) encrypts a block device with a master key that is itself protected by one or more passphrases or key files. Data is encrypted and decrypted on the fly, so applications see a normal filesystem.

For automatic unlock at boot, list the device in /etc/crypttab with a key file, or use network or TPM-based unlocking (Clevis with Tang, or TPM2) so a server can restart unattended without a person typing a passphrase.

Key facts: LUKS has several key slots, so you can add and remove passphrases without re-encrypting data. Back up the header, because if it is damaged the data is unrecoverable. Encryption protects data at rest (a stolen disk), not a running, unlocked machine. On CPUs with AES-NI the performance cost is small.

Read the drive's SMART health data with smartctl, watch the kernel log for I/O errors, and set up alerts through smartd or monitoring. Early warning signs show up in a few specific counters.

Attributes to watch on hard disks: Reallocated_Sector_Ct, Current_Pending_Sector and Offline_Uncorrectable. Rising values mean the disk is running out of spare sectors. On NVMe watch media_errors, percentage_used and critical_warning.

Caveats: a "PASSED" verdict does not guarantee health, since many disks fail with no warning. Disks behind hardware RAID need a controller option such as -d megaraid,N. Cloud virtual disks have no SMART data, so you rely on cloud metrics and filesystem errors instead. Combine SMART with RAID or replication, so one failing disk is an alert and not an outage.

VSZ is all the virtual address space a process has mapped (often huge and not real memory), RSS is the part currently in RAM, and PSS is RSS with shared pages divided fairly among the processes that share them. PSS is the fairest number for "how much memory does this process cost".

The trap: RSS counts shared libraries and shared memory fully in every process, so adding up the RSS of all processes can exceed the machine's RAM, which looks impossible. VSZ is even less useful, because memory-mapped files, reserved but unused address ranges and thread stacks inflate it. For a whole-machine view use MemAvailable in /proc/meminfo, not "free". Page cache is not charged to a process's RSS unless the file is mapped, which is why cache can use gigabytes without any process owning it.

Overcommit means the kernel lets programs allocate more virtual memory than the machine really has, betting that they will not use it all at once. It is why a large process can fork() successfully, and also why the OOM killer exists.

Allocation succeeds on paper, and the pages are only really taken when touched. If everyone then touches their memory, the kernel has no choice but to kill something.

Practical uses: Redis recommends mode 1, because its background save uses fork() and a strict check could make the fork fail on a big dataset. Mode 2 is safer for predictable systems, but it can break software (such as the JVM) that reserves large virtual ranges it never uses. Compare Committed_AS with CommitLimit to see how far you have promised beyond what exists.

Transparent Huge Pages (THP) let the kernel automatically use 2 MB memory pages instead of 4 KB pages, which reduces overhead for programs with large memory. The cost is occasional latency spikes and memory bloat, which hurts latency-sensitive databases.

Problems seen with Redis, MongoDB and others: the kernel may stall a thread to compact memory and build a huge page, causing random pauses of milliseconds or more. A background process (khugepaged) also merges pages, and copy-on-write of a 2 MB page after fork() can copy far more than needed, inflating memory use during a Redis save.

A common recommendation is madvise, where only programs that ask for huge pages get them, or never for databases whose vendors advise it. Do not confuse THP (automatic) with explicit HugePages (vm.nr_hugepages), which are reserved in advance and used deliberately by systems like Oracle and PostgreSQL.

A page fault happens when a process touches memory that is not yet mapped to a physical page. A minor fault is cheap, because the data is already in RAM and only the mapping is fixed, while a major fault needs a disk read and is thousands of times slower.

How to use it:

• Many minor faults are normal: first use of allocated memory, new processes, shared libraries.
• Many major faults mean memory-mapped files or swapped-out pages are being read from disk. After a restart (cold cache) or under memory pressure, a service can look slow for exactly this reason.
• If major faults rise together with swap activity (si and so in vmstat), the system is short of memory.

This explains why the first requests after a restart can be slow, and why a database that gets its cache evicted suddenly slows down. Fixes are more RAM, reducing memory use, pre-warming caches, or avoiding memory-mapping huge files you do not need.

A core dump is a snapshot of a process's memory and registers written when it crashes, so you can inspect the state at the moment of failure. You need it enabled, a place to store it, and a debugger.

Practical points: install debug symbols (debuginfo or -dbgsym packages) or the backtrace is just addresses. Core files can be as big as the process's memory and can fill a disk, so limit and rotate them. They contain everything in memory, including passwords and keys, so protect them. In containers, the kernel's core_pattern is global to the host, so crashes may not appear where you expect, and Kubernetes workloads need a plan for collecting them.

fork() creates a new process that is a copy of the caller, and exec() replaces that new process's program with a different one. The shell runs every command by doing fork and then exec.

You will see a clone (the modern fork) followed by execve("/bin/ls", ...).

Copying a whole process would be slow, so Linux uses copy-on-write (COW): parent and child share the same memory pages marked read-only, and a page is only copied when one of them writes to it. That makes fork() fast even for a process using many gigabytes.

Why it matters in practice:

• A shell can start thousands of short commands cheaply.
• Redis saves its dataset with fork(): the child sees a frozen snapshot while the parent keeps serving, but heavy writes during the save cause pages to be copied and memory use to grow.
• A fork() can still fail when memory is tightly limited (see overcommit).
• exec keeps the same PID and open file descriptors (unless marked close-on-exec), which is how redirections like > file reach the new program.

The program was built to use a shared library (.so file) that the dynamic linker cannot find at run time. The fix is to install the package that provides it or tell the linker where it lives, not to copy files around randomly.

The linker looks at RPATH/RUNPATH in the binary, LD_LIBRARY_PATH, the cache built by ldconfig (from /etc/ld.so.conf.d), then the default directories. Prefer a proper package or an ld.so.conf.d entry. Avoid setting LD_LIBRARY_PATH globally: it can make unrelated programs load the wrong library and is a security risk.

One more trap: ldd can actually execute the program in some cases, so do not run it on untrusted binaries. Use readelf -d or objdump -p instead.

systemd-analyze shows how long each boot stage took and which units were slowest, and critical-chain shows what the boot actually waited for. Start with the total, then drill down.

blame lists the time each unit took, but a slow unit that runs in parallel may not delay anything, so use critical-chain to see what really blocked the boot. Typical culprits:

• NetworkManager-wait-online or systemd-networkd-wait-online waiting for a network that is not there.
• An /etc/fstab entry for a missing or slow device (NFS, a removed disk). Add nofail and x-systemd.device-timeout=10s.
• cloud-init stages, or a slow DNS or time-sync step.
• A service with a long TimeoutStartSec.

Fix by disabling units you do not need, correcting dependencies, or adding timeouts, and measure again after each change.

systemd refuses to restart a service that fails too often in a short time. By default, more than 5 starts within 10 seconds puts the unit in a failed state with start-limit-hit, until you reset it.

Restart=on-failure restarts after a non-zero exit, a signal or a timeout, but not after a clean exit. Restart=always restarts even after a clean stop, which can hide a service that exits immediately by design. RestartSec adds a pause, which avoids hammering a dependency.

Raising StartLimitBurst only hides the problem. Read the journal and fix the cause, such as a bad config, a missing dependency, a wrong Type= (a daemon that forks started as simple), or a port already in use.

With socket activation, systemd opens the listening socket itself and starts the service only when the first connection arrives, handing the already-open socket to the process. The service does not need to be running first.

Benefits:

• On-demand start: rarely used services use no memory until needed.
• Zero-downtime restarts: while the service restarts, systemd keeps the socket open and queues incoming connections instead of refusing them.
• Parallel boot: other services can connect to the socket before the real service is ready.
• Privileged ports without root: systemd binds port 80 and passes it to an unprivileged process.

The application must support receiving the socket (using the LISTEN_FDS convention, via a systemd library or a framework). sshd, Cockpit and some container tools use this pattern.

systemd can restrict a service with simple directives that use namespaces, capabilities and system call filters, so a compromised service can reach far less of the system. systemd-analyze security scores how exposed a unit is.

Add restrictions one at a time and restart, because each one can break an application that quietly writes to an unexpected path or needs a certain capability. Read the journal after each change to see what was denied. This gives you much of the isolation benefit of a container, with no image or runtime, which is valuable for services you run directly on the host.

Turn on IP forwarding so the kernel passes packets between interfaces, allow the traffic in the firewall's FORWARD chain, and add a masquerade (source NAT) rule on the outgoing interface so private addresses are hidden behind the gateway's address.

How it works: a client sends a packet to the gateway, the gateway forwards it out and rewrites the source address to its own, and the connection tracking table (conntrack) remembers the mapping so replies are translated back.

Common failures: forwarding disabled or not made permanent, a default-DROP FORWARD policy with no allow rule, missing return-traffic rule, clients using the wrong default gateway, and rules lost at reboot because they were never saved. Check with ip route, sysctl net.ipv4.ip_forward, conntrack -L and tcpdump on both interfaces.

Every packet passes through hooks in a fixed order, and at each hook the matching chain's rules are checked top to bottom; the first rule that matches and has a final action (ACCEPT, DROP) decides. If no rule matches, the chain's default policy applies.

Key facts:

• Incoming traffic for the server itself uses INPUT, traffic passing through uses FORWARD, and traffic the server creates uses OUTPUT.
• NAT happens in PREROUTING (destination NAT) and POSTROUTING (source NAT).
• Order matters: a broad DROP above a specific ACCEPT blocks it. -I inserts at the top and -A appends to the bottom.
• Connection tracking lets you write one rule for return traffic: -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT.

Counters are a great debugging tool: a rule whose counter does not rise is not matching. Container tools such as Docker add their own chains, which can surprise you.

Connect with openssl s_client to see exactly what certificate and chain the server sends, then check dates, names and the chain. This shows what a client sees, regardless of what a browser hides.

Common causes:

• Expired certificate, or the system clock on the client is wrong.
• Missing intermediate certificate: browsers can fetch it, but curl and many libraries cannot, so "works in the browser, fails in my script". The fix is to serve the full chain on the server.
• Hostname mismatch: the name is not in the certificate's Subject Alternative Names.
• SNI: without -servername, a server hosting many sites may return the wrong certificate.
• Old or missing CA bundle on the client (update-ca-certificates or update-ca-trust).

Never "fix" it with curl -k in production code, since that turns off verification and removes the protection TLS provides.

The command resolves the name, finds the route, opens a TCP connection, performs a TLS handshake, then sends the HTTP request and reads the response. Each step has its own failure mode and its own tool.

curl can time each stage for you:

Read the numbers as cumulative: a large dns points to resolver trouble (getent hosts, /etc/resolv.conf), a large connect to routing or packet loss (ip route get, mtr), a large gap before tls to a slow handshake or an overloaded server, and a large firstbyte to a slow application. Then confirm with tcpdump, openssl s_client and the server logs. Knowing this flow is the best answer to the classic question "what happens when you type a URL".

A bridge works like a virtual switch that joins interfaces into one network, a bond combines several physical NICs into one logical link, and a VLAN interface carries one tagged virtual network over a physical link.

• Bridges: VM hypervisors (KVM) and container networks (docker0) use them to connect guests to the LAN.
• Bonds: active-backup gives failover with no switch support, and 802.3ad (LACP) aggregates links but needs matching switch configuration.
• VLANs: separate traffic types (management, storage, tenants) on one cable. The switch port must be a trunk that allows those tags.

A trap: LACP spreads traffic per flow, so one single TCP connection does not go faster than one NIC, and only many connections together use both links. Commands like these are temporary, so make them permanent through Netplan, NetworkManager or systemd-networkd.

A hard mount keeps retrying forever if the server stops answering, so any process touching it freezes (often in D state). A soft mount gives up after retries and returns an error, but risks data corruption for writes. "Stale file handle" means the client holds a reference to a file or export that no longer exists on the server.

Stale handles appear when the exported directory is deleted and recreated, the server's filesystem is restored or the export changes. The fix is to remount on the client. Hangs usually mean the server or network is unreachable, so check connectivity first. Use hard for data you care about, and design applications so they tolerate a slow mount. Automount (x-systemd.automount) keeps a dead NFS server from blocking boot.

Forward logs from every host to a central collector over a reliable, encrypted channel, with local queues so nothing is lost when the collector is down. Do not rely on UDP syslog for important logs.

Alternatives include systemd-journal-remote and agents such as Fluent Bit, Vector or Promtail that ship to a log platform.

Common problems:

• UDP loss: @ means UDP in rsyslog, @@ means TCP. UDP drops messages silently.
• No queue: logs are lost during outages unless a disk-assisted queue is configured.
• Rate limiting: journald and rsyslog drop bursts by default (RateLimitBurst), which hides the very flood you want to see.
• Time: use UTC and NTP, or events cannot be lined up across hosts.
• Parsing: use structured (JSON) logs where possible, and protect against log injection and sensitive data in messages.
• Capacity: plan retention and disk size at the collector.

Check each resource for utilisation, saturation and errors (the USE method), plus a few things that fail silently, and alert on what hurts users rather than on every raw metric. Too many alerts teach people to ignore them.

Alert on trends for disk, not just thresholds, since "full in 6 hours" is more useful than "80% full". For example in Prometheus with node_exporter:

Use MemAvailable, not "free" memory, because Linux uses spare RAM for cache. Combine host metrics with application symptoms (error rate, latency) so you can tell a real incident from a harmless spike. Review and delete alerts that nobody acts on.

A production script fails loudly, cleans up after itself, cannot run twice at once, and can be tested without causing damage. The goal is predictable behaviour when something goes wrong, not clever code.

Checklist behind it: strict mode, quoted variables, mktemp plus trap, a lock, input validation, a dry-run switch, timestamped logs to stderr, meaningful exit codes, absolute paths or an explicit PATH, and retries with a limit for flaky network calls. Also make steps idempotent, so a rerun after a half-failure is safe. Finally, lint with ShellCheck in CI and move to Python or a real tool once the script needs complex data handling.

Give people named accounts, grant access by role through groups, allow only exact commands, log everything, and keep a separate break-glass path for emergencies. Never hand out blanket ALL rights to a whole team.

Design points:

• Exact commands, absolute paths. systemctl restart * would allow restarting any service. Avoid wildcards, shells, editors and interpreters (see shell escapes).
• Roles in config management. Keep sudoers files in Ansible or similar, validate with visudo -c, and review changes like code.
• Logging. sudo logs to syslog; add log_output for session recording and forward logs off the host.
• Temporary access. Use short-lived, approved grants (an access tool or a timed config) rather than permanent membership.
• Break-glass. A rarely used root credential kept in a vault, alerting on use.
• No NOPASSWD for humans. Use it only for narrowly scoped automation accounts.

The trade-off is friction versus control. Start strict for production and relax only with a documented reason.

Use a proper orchestration tool (Ansible, pssh, or your fleet manager) with inventory, timeouts and batching, try it on a canary first, and stop automatically if failures pile up. A plain for host in ...; do ssh loop is slow, serial and has no safety net.

Plan it like this:

1. Make the command idempotent and test it on a staging host and with a dry run (--check --diff).
2. Canary first, then growing batches, checking health between batches.
3. Set timeouts (ConnectTimeout, command timeouts) so one dead host does not stall the run.
4. Limit blast radius with a maximum failure percentage and a clear stop condition.
5. Collect results per host and review failures, instead of scrolling terminal output.
6. Take the host out of the load balancer if the action causes downtime, and have a rollback.

The common mistake is running a risky command everywhere at once because "it worked on my machine". A staged rollout turns a potential outage into one failed batch.

Make internal servers reachable only through a hardened bastion, and have a certificate authority (CA) sign users' public keys for a short time. Servers trust the CA, so there are no authorized_keys files to maintain and access expires by itself.

Why this is better at scale:

• Expiry built in. A leaked certificate stops working in an hour, while a leaked public key in authorized_keys works until someone finds it.
• No key sprawl. Joiners and leavers do not need changes on every server.
• Principals map a certificate to allowed login names, so one cert can carry roles.

For the bastion itself: only SSH open to the internet, key or certificate login only, MFA at the CA step, ProxyJump for users, no agent forwarding, LogLevel VERBOSE plus session logging, and internal servers allowing SSH only from the bastion's address. The weak point becomes the CA private key, so protect it in an HSM or a vault and audit every signing.

Keep secrets out of command lines and images, hand them to services as protected files or from a secrets manager, and use short-lived credentials. Environment variables and process arguments are more visible than most people expect.

How they leak:

• Arguments: mytool --password=abc is visible to every user in ps and in /proc/<PID>/cmdline.
• Environment: readable by the same user and root in /proc/<PID>/environ, inherited by every child process, and often dumped in crash reports and debug pages.
• History and logs: shell history, CI logs, and set -x traces.

Better options:

At scale, use a secrets manager (Vault, a cloud KMS or secret store) with short-lived, rotated credentials and audit logs. Never bake secrets into container images or Git. Treat any secret that appeared in a log or command line as leaked and rotate it.

Attackers who get a low-privilege shell look for misconfigurations that turn it into root. Most paths are about too many privileges, writable places that root trusts, or missing patches.

Common paths and fixes:

Defences work in layers: least privilege, patching, filesystem mount options (nosuid), MAC (SELinux or AppArmor), auditing with auditd, and regular scans with tools like lynis. A useful habit is to run an enumeration check yourself (sudo -l, find for SUID and writable paths) before an attacker does.

Assume a root attacker can edit or delete anything on the machine, so logs that exist only locally cannot be trusted. Ship them off the host in near real time to a system the attacker cannot reach.

Layers to combine:

1. Remote, append-only storage. Forward with rsyslog over TLS, systemd-journal-remote or an agent such as Fluent Bit or Vector to a central store with restricted write access and retention locks.
2. Local hardening. chattr +a on important log files, tight permissions, and auditd in immutable mode (-e 2) so rules cannot be changed without a reboot.
3. Tamper evidence. journald Forward Secure Sealing, or hash chaining and signing in your SIEM, so edits can be detected.
4. Alerts on the logging path itself. Alert on missing heartbeats, rsyslog or auditd stopping, log files being truncated, and time jumps.
5. Accurate time. NTP everywhere, because an attacker can also change the clock to confuse timelines.

The key idea: a gap in logs is itself evidence. Central logging with alerts on silence turns "the logs were deleted" into "we know exactly when they stopped".

Put hard limits on CPU, memory, processes and disk per user with cgroups (through systemd) and quotas, and add isolation for temporary files. Soft habits like nice help, but they do not give a guarantee.

Other controls:

• Per-process limits in /etc/security/limits.conf (applied by pam_limits): nproc, nofile, core.
• Disk quotas: quotaon for ext4, and XFS project quotas for directories.
• Isolated temp space: pam_namespace for per-user /tmp, or PrivateTmp=yes for services.
• Priority: nice and ionice for background work.
• Visibility: systemd-cgtop, loginctl, and alerts on per-user resource use.

Use cgroup limits for fairness, because ulimit can be bypassed by launching many processes. The trade-off is that a memory limit that is too tight makes the kernel kill the user's legitimate job, so set limits from observed usage and communicate them.

Stream the file through awk and sort, keep only small summaries in memory, and set LC_ALL=C for speed. This works as long as the number of distinct keys stays reasonable. Assume an nginx log where the client IP is field 1, the time is field 4 and the request time in seconds is the last field.

Why it works: each tool reads a stream, so memory use is small. The percentile needs a sorted list, so sorting is the heavy step, and GNU sort spills to temporary disk, so point it at a big disk with sort -T /bigdisk. Trade-offs: awk associative arrays grow with unique keys (millions of unique IPs can be heavy), and for repeated analysis, load the data into a proper tool (a database, ClickHouse, or your log platform) instead of re-scanning 50 GB each time.

Prepare the new version in a separate directory, then switch to it with one atomic operation: a rename. Never copy files over a live directory.

Key points:

• rename() (what mv uses) is atomic only inside one filesystem. Staging in the same filesystem is essential.
• ln -sfn new current is not atomic, because it removes the old link and then creates the new one. Create a temporary link and mv -T it over the old one instead.
• Keep several old releases, so rollback is the same switch pointed backwards.
• Running processes keep their old files open, so reload or restart is still needed.
• For single files, write to a temp file in the same directory, flush it, then mv it into place.

Trade-offs: symlink swaps need the app to resolve paths through the link each time, and shared state (uploads, sockets) must live outside the release directories. Database migrations are not covered by file atomicity, so use backward-compatible migrations.

Build the new server from code, copy the data in repeated passes while the old one is still serving, then do a short final sync during a brief write freeze and switch traffic. Keep the old server ready for rollback.

Plan:

1. Lower the DNS TTL a day or more ahead (for example to 60 seconds), so the cutover takes effect quickly.
2. Build the target with configuration management, with the same packages, users, cron jobs, certificates and firewall rules. Check for hard-coded IP addresses and local-only secrets.
3. Bulk copy while live:
   Bash

   rsync -aHAX --numeric-ids --info=progress2 old:/srv/data/ /srv/data/
4. Repeat the sync to shrink the delta. Replicate databases with native replication instead of file copies.
5. Cutover: put the old service in maintenance or read-only mode, run the last sync, verify (row counts, checksums, health checks), then switch DNS or the load balancer.
6. Rollback path: keep the old server running, and plan a reverse sync if writes landed on the new one.
7. Watch error rates and logs closely, and only decommission after a safe period.

The trap is forgetting things that are not data: scheduled jobs, TLS certificates, monitoring, backups, and firewall allowlists on other systems that trust the old IP. A written checklist and a rehearsal on a copy prevent most surprises.

Idempotent means running the same automation repeatedly gives the same final state and changes nothing the second time. Drift is when a server no longer matches its intended configuration, usually because of manual changes.

In plain shell, write "check, then change":

Declarative tools such as Ansible, Puppet or Chef describe the desired state (state: present) and only act on differences. They also report what would change:

To control drift: run the configuration tool on a schedule and alert when it reports changes, restrict manual root access, and keep configuration in Git so every change is reviewed. Handlers should run only when something changed (restart a service only after its config changed). The alternative philosophy is immutable servers: never fix in place, rebuild from a new image, which removes drift by design. Test automation in disposable containers or VMs so you know a rerun is truly safe.

Stop writes to that filesystem immediately, because every new write can overwrite the freed blocks. Then recover from the safest source available, starting with backups and snapshots, and only then try risky undelete tools.

Order of attack:

1. Stop the damage. Stop the application, and remount read-only or unmount if possible.
2. Is a process still holding the file open? Then the data is still there:
   Bash

   lsof +L1 # deleted files still open

   cp /proc/<PID>/fd/<FD> /safe/place/recovered.dat
3. Snapshots and backups. LVM, ZFS, btrfs or cloud volume snapshots, then the last backup. This is the reliable path.
4. Undelete tools as a last resort. On ext4, tools like extundelete or debugfs sometimes work; XFS has no practical undelete. testdisk and photorec carve file contents without names. Always work on an image copy (ddrescue), never on the original disk. SSD TRIM and discard options often erase deleted data quickly, which lowers the odds a lot.

Prevention: tested backups and regular snapshots, least privilege so few people can run recursive deletes as root, quoting and set -u in scripts, using ls or echo to preview, and separate production and staging terminals with clearly different prompts. The honest message to management is that after rm the odds are poor, which is why backups you have actually restored are the real protection.

D means uninterruptible sleep: the process is inside the kernel waiting for I/O and cannot receive signals until that I/O returns. So the real problem is whatever it is waiting on, usually storage or a network filesystem, not the process itself.

Typical causes and responses:

• NFS server or network gone, with a hard mount. Restore the server or network, or umount -f / umount -l the share.
• Failing disk, controller, or SAN path. Look for I/O errors, check SMART and multipath, replace or reconnect.
• Hung FUSE filesystem or buggy driver. Stop the userspace helper or reload the module.
• Storage throttled in the cloud. Check volume metrics.

Newer kernels let some network waits be killable, which is why kill -9 sometimes works on NFS and sometimes not. Fix the cause of the stall, because rebooting is the last resort that also loses the evidence. Remember that many D-state tasks also raise the load average, which is why high load with idle CPU is a clue.

When the kernel itself fails, normal logs may never be written, so you prepare beforehand: a crash kernel (kdump) to save memory, a serial or cloud console to see messages, and settings that make failures observable instead of silent.

After a crash, kdump boots a small second kernel and writes a vmcore file (often under /var/crash). Analyse it with the crash tool:

Other tools:

• Magic SysRq (echo 1 > /proc/sys/kernel/sysrq): keys such as w (blocked tasks), t (all tasks), s u b (sync, remount read-only, reboot). To test kdump you can use echo c > /proc/sysrq-trigger, but this crashes the machine deliberately, so do it only on a test host.
• Serial or cloud console with console=ttyS0 on the kernel line, to read messages that never reach disk.
• Hardware watchdog to reset a hung machine, and out-of-band management (iDRAC, iLO) for physical hosts.

Test the whole path before you need it, because a kdump that was never tested often fails when it matters.

Linux buffers writes in the page cache as "dirty" pages and flushes them to disk in the background. When too many pages are dirty, the kernel makes writing processes wait until enough has been flushed, and your application stalls.

How it works: when dirty data passes vm.dirty_background_ratio (default 10% of available memory), background flushing starts. When it passes vm.dirty_ratio (default 20%), writing processes are throttled and block. On a machine with a lot of RAM and a slow disk, 20% can be many gigabytes, so the kernel accumulates a huge backlog and then stalls everything while it drains.

Options:

• Set byte-based limits that suit the disk speed (for example background 256 MB, hard limit 1 GB) with vm.dirty_background_bytes and vm.dirty_bytes.
• Use faster or provisioned storage, and spread writes over time.
• Make the application write steadily, and call fsync in small batches, not one huge flush.
• Use ionice or cgroup I/O limits for bulk jobs, so they do not starve latency-sensitive writers.

Measure before and after, since the right values depend on disk throughput.

Yes. A CPU limit (a cgroup quota) lets the container use only a fixed amount of CPU time per period, for example 200 ms in every 100 ms window for 2 cores. A multi-threaded app can burn its whole quota in a few milliseconds, then be paused until the next period, even when the host has plenty of free CPU.

In cpu.stat, a growing nr_throttled (and throttled_usec) next to nr_periods proves throttling. In Kubernetes the same signal is container_cpu_cfs_throttled_periods_total divided by total periods.

Causes and fixes:

• Limit too low for bursty work. Raise or remove the CPU limit, but keep a request so scheduling stays fair.
• Many threads versus a small quota. Runtimes may see the host's cores, not the quota (older Go and JVM versions), so match thread pools to the limit (GOMAXPROCS, JVM container support).
• Burst control. cgroup v2 offers cpu.max.burst on newer kernels, which lets short bursts borrow unused quota.

The lesson for monitoring: average CPU usage hides throttling, so graph throttled time alongside latency. A container at 40% average CPU can still be heavily throttled.

A network namespace is an isolated copy of the network stack, with its own interfaces, routes, firewall rules and ports. A veth pair is a virtual cable with two ends: whatever enters one end comes out the other, so you can place one end inside the namespace and keep the other outside.

A container runtime does this for every container: it creates a namespace, creates a veth pair, attaches the host end to a bridge (like docker0), gives the container end an IP, and adds NAT (masquerade) rules so containers can reach the outside.

Debugging tips:

nsenter is very useful when the container image has no network tools. Typical failures are a missing forwarding rule or NAT rule, a firewall chain ordering problem, or overlapping IP ranges between the container network and the host network.

Each listening socket has two queues: a SYN queue for half-open connections, and an accept queue for completed handshakes waiting for the application to call accept(). If the accept queue is full because the application is too slow, new connections are dropped or delayed.

How to read it: if Recv-Q is close to Send-Q on a LISTEN socket, or ListenOverflows keeps rising, the application is not accepting connections fast enough. The queue size is the smaller of the application's listen(backlog) value and net.core.somaxconn.

Fixes, in order of importance:

1. Find why the app accepts slowly: an exhausted thread pool, a blocked event loop, GC pauses or CPU saturation on one core.
2. Raise the backlog in the application (for example nginx listen ... backlog=) and somaxconn.
3. Scale out or add capacity.

SYN cookies (tcp_syncookies) protect against SYN floods by avoiding state, but they do not fix a slow application. To tell a legitimate surge from an attack, look at the source addresses and whether handshakes complete (tcpdump, ss -s). Raising limits alone just moves the queue and increases latency.

If a packet arrives on one interface but the server's route to the sender goes out a different interface, reverse path filtering (rp_filter) may drop it as spoofed. Even without it, stateful firewalls on one path may drop replies that took a different path.

Typical setup: a server with two NICs (public and private) and one default gateway. A client reaches the private address, but the reply leaves through the public NIC by the default route, so the client or a firewall drops it.

Fixes:

• Policy routing: make replies leave the interface they came in on.
  Bash

  ip route add default via 10.0.1.1 dev eth1 table 100

  ip rule add from 10.0.1.10 table 100
• Loose rp_filter (2) where asymmetric paths are legitimate, instead of turning it off completely.
• Make sure firewalls see both directions of a flow.

The clue is that tcpdump shows the SYN arriving but no useful reply on the same path, or a reply on a different interface. This is a classic issue on cloud instances with several network interfaces.

Grow the volume in the cloud console or API first, then extend the partition, then any LVM layer, then the filesystem. Each layer must be told about the new size, in that order, and a snapshot comes first.

Checks and cautions:

• Snapshot or backup first, because a mistake in partition editing can lose data.
• Growing is online and safe for ext4 and XFS, but shrinking is not possible for XFS and is risky for ext4.
• Some cloud providers limit how often a volume can be modified, so plan sizes with headroom.
• On GPT disks the backup header must move to the new end, and growpart handles that.
• Many cloud images grow the root filesystem automatically at boot through cloud-init, but you may need to do it by hand on a running system.

Also find out why the disk filled up (logs, caches, df -i), or you will repeat this in a month.

Identify the failed member, remove it, add a new disk that is partitioned the same way, and let the array rebuild while you watch it. The risky part is the rebuild window, when the array has no redundancy left.

Risks and good practice:

• Second failure during rebuild. On RAID 5, an unreadable sector on a surviving disk can fail the rebuild. Larger disks make this more likely, so prefer RAID 6 or RAID 10 for big arrays.
• Back up first if you can, since a rebuild is heavy I/O on old, stressed disks.
• Use the right device, and double-check serial numbers before pulling a disk.
• Keep a hot spare so rebuild starts automatically, and enable monitoring (mdadm --monitor, mail alerts) so you hear about the first failure at once.
• Update mdadm.conf and the initramfs when the layout changes, and tune rebuild speed with dev.raid.speed_limit_min if the rebuild is too slow.

RAID is not a backup: it protects against disk failure, not deletion, corruption or ransomware.

Start from what the business needs (how much data loss and downtime is acceptable), choose methods that meet it, keep copies in separate failure domains, and prove that restores work. A backup you have never restored is only a hope.

Key terms: RPO (recovery point objective, how much data you can lose) and RTO (recovery time objective, how long recovery may take).

Design points:

• 3-2-1 rule: three copies, on two kinds of storage, with one off-site. Add an immutable or offline copy for ransomware protection.
• Method per data type: file backups with deduplication (restic, borg), database-native backups (pg_dump, binary backups plus WAL), and volume or cloud snapshots for fast whole-machine recovery.
• Consistency: databases need a consistent state (native tools, or fsfreeze with snapshots), otherwise the backup may not restore.
• Security: encrypt backups, keep keys separate from the backup store, and use credentials that cannot delete backups.
• Monitoring: alert on failed or missing jobs, not just on success messages.
• Retention and cost: daily, weekly and monthly policies.

Schedule restore drills (at least a sampled file restore monthly, a full rebuild a few times a year) and write the recovery steps down. Servers that can be rebuilt from code need only their data backed up, which shrinks RTO.

Treat it as a staged rollout: test, canary, small batches, with a tested way back. A kernel change touches drivers, networking and storage, so surprises show up only after the reboot.

Plan:

1. Read the changes and test on a staging host with the same hardware or instance type and workload.
2. Check dependencies that build against the kernel, such as DKMS modules (NVIDIA, ZFS, some network drivers): dkms status. A missing module can leave the server without its network or storage after reboot.
3. Know what is pending. Compare the running kernel with the installed one.
   Bash

   uname -r # running kernel

   needs-restarting -r # RHEL: is a reboot needed?

   ls /var/run/reboot-required # Ubuntu: exists when a reboot is needed
4. Keep the old kernel in the boot menu, and use a one-time boot test where possible:
   Bash

   sudo grub-reboot "Advanced options for Ubuntu>Ubuntu, with Linux 6.8.0-45-generic"

   sudo grubby --default-kernel # RHEL: show the default
5. Roll out in stages: one canary, then small batches. Drain each host from the load balancer first, reboot, then verify (systemctl --failed, mounts, network, application health) before continuing.
6. Have a rollback: boot the previous kernel, and stop the rollout automatically on failures.

For urgent security fixes, live patching (kpatch, Canonical Livepatch) can reduce emergency reboots, but it covers only some fixes, so schedule normal reboots too. Make sure the console (cloud serial console or iDRAC) works before you start, because a kernel that does not boot needs it.

Averages hide short stalls, so look at where time is spent per request: running on the CPU, waiting for the CPU, or blocked (off-CPU) on I/O, locks or the network. Modern tracing tools show this with low overhead.

Approach, from cheap to deep:

• Look at distributions, not averages. Use percentiles and histograms (p99), and per-process numbers: pidstat -u -d -w 1 (CPU, disk, context switches).
• Is the process waiting to run? runqlat (bcc) shows run-queue latency, which reveals CPU contention or cgroup throttling.
• Is it blocked? offcputime shows where threads sleep, and biolatency shows disk I/O latency histograms.
• Network: tcpretrans shows retransmits, and ss -ti shows RTT per connection.
• What system calls are slow? strace -T -tt -p PID timestamps each call (this slows the process a lot, so use it briefly or on a canary).
• Where is CPU going? perf record -g -p PID -- sleep 30, then perf report or a flame graph.
• Short-lived processes: execsnoop reveals commands that start and exit too fast to see in top.

eBPF tools (bcc, bpftrace) run in the kernel with low overhead and need a recent kernel, root access and often matching kernel headers. Be careful in production: always know a tool's overhead, since strace can slow a process by an order of magnitude while perf sampling is light. The point to make aloud is the method: measure where time goes, form a hypothesis, then confirm it with a second tool.

The first process in a container is PID 1 in its own PID namespace, and Linux treats PID 1 specially. A normal program as PID 1 does not get the default signal behaviour, and it also becomes the parent that must clean up orphaned processes.

Two consequences:

1. Signals. The kernel does not apply the default action (terminate) to PID 1 for signals it has not installed a handler for. If your app never handles SIGTERM, it simply ignores it. docker stop sends SIGTERM, waits 10 seconds, then sends SIGKILL, which is why shutdowns are slow and unclean.
2. Zombies. Orphaned child processes are re-parented to PID 1. If PID 1 never calls wait(), finished children stay as zombies and slowly fill the process table.

Shell scripts make it worse: if the entrypoint is a shell script that starts the app as a child, the shell receives SIGTERM and does not pass it on.

Fixes:

You can also bake in tini or dumb-init, make the application handle SIGTERM (graceful shutdown), and set STOPSIGNAL if it expects a different signal. Kubernetes behaves the same way, so a good terminationGracePeriodSeconds plus proper signal handling avoids dropped requests during deploys.