It is 3 AM. Hotstar's streaming service is throwing 503s for 2 million concurrent users during an IPL match. Your on-call rotation fires. You SSH in and run `systemctl status streaming-api` - it says `active (running)`. You run `top` - CPU looks fine. You run `free -h` - memory looks fine. Everything looks fine. But users cannot stream. The actual problem: a single upstream payment microservice had leaked 60,000 open file descriptors. The kernel's per-process FD limit was hit. New incoming TCP connections from the CDN were silently failing because the process could not call `accept()` - no file descriptor left to represent the new socket. The service was running perfectly. It just could not accept new work. `systemctl` could not tell you this. `top` could not tell you this. Only reading `/proc/<pid>/fd/` and knowing what you were looking at could tell you this - in under 30 seconds, during a live P0. This module teaches you what is happening inside the kernel that every surface-level tool hides from you. You already know how to use `systemctl`, `ps`, `top`, and `kill` from the Linux Fundamentals module. This module goes underneath those tools - to the syscall level, the memory page level, the cgroup boundary level - so that when the tools lie to you at 3 AM, you still know exactly where to look.
Before a single process runs, the kernel has to get off the ground. Understanding boot is not academic - it is how you diagnose a server that will not come back after a reboot, a kernel panic with no logs, or a service that does not start even though its binary exists. ### What actually happens between power-on and your first prompt Think of boot as a relay race. Each stage has one job, and if it drops the baton, the next stage never runs. The failure mode at each stage is completely different. Stage 1: BIOS / UEFI Power on -> firmware runs from ROM chip on motherboard Checks: CPU, RAM slots, storage detected Finds: the first bootable disk (MBR or GPT partition table) Hands off to: the bootloader stored in the first 512 bytes of disk Stage 2: GRUB (Grand Unified Bootloader) Reads its config from /boot/grub/grub.cfg Shows the boot menu (or skips it after timeout) Loads: the kernel image (vmlinuz) into RAM Loads: initramfs - a tiny temporary filesystem in RAM Hands off to: the kernel Stage 3: Kernel Init Decompresses itself (vmlinuz is a compressed archive) Detects all hardware: CPU cores, RAM, storage controllers Mounts the initramfs as temporary root (/) Runs scripts from initramfs to find and mount the REAL root disk Pivots root to the real disk Runs: /sbin/init as PID 1 Stage 4: systemd (PID 1) The very first real process - parent of everything Reads /etc/systemd/system/ unit files Starts services in dependency order Gives you the login prompt when the default.target is reached **What breaks at each stage and how to diagnose it:** | Stage | Failure Symptom | Diagnosis Tool | | :--- | :--- | :--- | | BIOS/UEFI | No POST, black screen, beep codes | Physical access, BIOS console | | GRUB | Grub rescue prompt, file not found | Boot from rescue media, `grub-install` | | Kernel init | Kernel panic: unable to mount root | Boot older kernel from GRUB menu | | initramfs | Drops to initramfs shell | `lsinitramfs /boot/initrd.img` | | systemd | Services fail, no network | `journalctl -b` for boot logs | ```bash ## Read all logs from the most recent boot journalctl -b ## Read logs from the PREVIOUS boot - critical when server crashed and rebooted journalctl -b -1 ## See why a specific service failed to start during boot journalctl -b -u nginx --no-pager ## Check what systemd target the system booted into systemctl get-default ## Output: multi-user.target (server) or graphical.target (desktop) ## See the boot time breakdown - which services took longest systemd-analyze blame | head -20 ## Full critical chain - the bottleneck in your boot time systemd-analyze critical-chain ``` ### Why systemd as PID 1 matters for SRE work **PID 1** is not just the first process - it has a special role in the Unix process model. Every process has a parent. When a parent process dies, its children become orphans. The kernel reassigns orphaned processes to PID 1. If PID 1 dies, the kernel panics - there is no recovery path. This means systemd is responsible for reaping zombie processes from the entire system, not just its own children. It is also responsible for the signal behaviour of every service it manages - if you send SIGTERM to a service, systemd is the one that sends it to the process and waits for it to exit cleanly before declaring it stopped. ```bash ## PID 1 is always systemd on modern Linux ps -p 1 -o pid,ppid,cmd ## Output: ## PID PPID CMD ## 1 0 /sbin/init ## Everything else has PID 1 somewhere in its ancestor chain pstree -p | head -30 ``` > 📌 **Remember:** If a container has PID 1 = your application (not an init system), and your app crashes, the container exits immediately. There is no one to reap zombie child processes either. This is why production containers should use `tini` or `dumb-init` as PID 1, not your app directly.
You already know processes exist. This section explains how they are born, how they communicate, and how they die at the kernel level - because most production process problems (zombie processes, defunct entries in `ps`, signal delivery failures) are invisible unless you understand this layer. ### How fork and exec actually create a process Every process in Linux is created by another process using two syscalls working together: `fork()` and `exec()`. There is no other way. Think of it like this: imagine a photocopier that can copy itself. `fork()` is the copy operation - it produces an identical duplicate. `exec()` is what happens next - the copy tears itself apart and rebuilds as something completely different. fork() - Create a near-identical copy of the current process | +-> Parent process continues running with original code | Parent gets: return value = child's PID | +-> Child process starts running the same code Child gets: return value = 0 (this is how it knows it is the child) Child has: copy of parent's memory, same open files, same env vars Child does: calls exec() to replace itself with the new program exec() - Replace the current process image with a new program | +-> Kernel loads the new binary from disk into memory +-> Old code and data are completely replaced +-> New program starts executing from its entry point +-> Open file descriptors are preserved (unless marked close-on-exec) ```bash ## Watch these syscalls happen in real time with strace ## Run strace on bash itself - see it fork and exec when you run a command strace -f -e trace=clone,execve bash 2>&1 | head -50 ## The -f flag follows child processes (fork) ## clone() is the modern Linux equivalent of fork() ## execve() is the exec syscall ## Simpler: trace just the fork+exec of a single command strace -e trace=clone,execve ls /tmp 2>&1 ## You will see: execve("/bin/ls", ["ls", "/tmp"], ...) = 0 ``` > **Note:** `strace` attaches to a process and intercepts every syscall it makes, showing you the arguments and return values. It is the X-ray of Linux process debugging - it shows you exactly what a process is asking the kernel to do. ### Zombie processes - the undead that haunt production A **zombie process** is a process that has finished executing but has not been cleaned up by its parent yet. It is dead - it uses no CPU, no memory, no resources - but it still occupies a slot in the process table and has a PID reserved. How does this happen? When a process exits, it leaves behind an exit status code - a number saying how it finished (success = 0, various failures = non-zero). The parent process is supposed to call `wait()` to collect this exit code. Until the parent calls `wait()`, the dead process stays in the process table as a zombie. Child process exits | Kernel marks it ZOMBIE (Z state in ps output) Kernel holds: exit code, PID, resource usage stats Kernel waits for parent to call wait() | Parent calls wait() <-- this MUST happen | Kernel releases the process table entry PID is freed, zombie is gone The problem is when a parent **never calls `wait()`** - either because it is poorly written, or because it is so overloaded it cannot keep up with dying children. ```bash ## Spot zombies - look for Z in the STAT column ps aux | grep 'Z' ## Or look for [defunct] in the command column ps aux | awk '$8 == "Z" {print $0}' ## Example zombie output: ## USER PID %CPU %MEM STAT COMMAND ## nobody 4821 0.0 0.0 Z [java] <defunct> ## Find the parent of a zombie (the one that needs to call wait()) ## Get the PPID of the zombie process ps -o ppid= -p 4821 ## Output: 4800 <- this is the parent PID ## Check what the parent is doing ps -p 4800 -o pid,ppid,stat,cmd ## How many zombies exist right now? ps aux | awk '$8 == "Z"' | wc -l ``` **Why zombies matter in production:** Each zombie holds one slot in the process table. The process table has a finite limit (usually 32768 PIDs by default). A poorly written Java application spawning threads that become zombies can exhaust the PID space. New processes - including health check scripts, log rotation jobs, monitoring agents - will fail to spawn. The system appears to lock up even though CPU and memory are fine. > 🔴 **Common Mistake:** People try to kill zombie processes with `kill -9`. This does nothing - the zombie is already dead. The fix is to kill or fix the parent process so it calls `wait()`. Once the parent calls `wait()` or the parent dies (after which PID 1 reaps the zombies), the zombies disappear. ### Orphan processes - when parents die first An **orphan process** is the opposite situation: the parent dies while the child is still running. The child is not dead - it is actively running - but it has no parent. The kernel immediately reparents orphans to PID 1 (systemd). Systemd will call `wait()` when they eventually die. Orphans are not a problem in themselves - they keep running fine. The issue is that they are now disconnected from any session or terminal, which means they will not receive terminal signals like SIGHUP when you close your SSH session. This is exactly why `nohup` and `disown` exist - they intentionally orphan your background processes so they survive SSH disconnection. ```bash ## See the parent-child relationships in tree form pstree -p ## Find all processes whose parent is PID 1 (reparented orphans OR legitimate daemons) ps -eo pid,ppid,cmd | awk '$2 == 1 {print}' ## Check the PPID of any process ps -o pid,ppid,cmd -p <PID> ``` ### Signals - how processes talk to each other **Signals** are the kernel's mechanism for sending notifications to processes. They are asynchronous - a process can receive a signal at any point during execution, even in the middle of another operation. The most important signals for SRE work: | Signal | Number | Default action | Meaning | | :--- | :--- | :--- | :--- | | SIGHUP | 1 | Terminate | Terminal closed, or "reload config" by convention | | SIGINT | 2 | Terminate | Ctrl+C from keyboard | | SIGTERM | 15 | Terminate | Polite shutdown request - can be caught and handled | | SIGKILL | 9 | Kill immediately | Cannot be caught or ignored - kernel kills directly | | SIGSTOP | 19 | Suspend | Cannot be caught - kernel suspends the process | | SIGCONT | 18 | Resume | Resume a stopped process | | SIGCHLD | 17 | Ignore | Child process died - parent receives this | ```bash ## Send SIGTERM (polite - process can clean up) kill 1234 kill -15 1234 kill -TERM 1234 ## Send SIGKILL (instant - no cleanup possible) kill -9 1234 kill -KILL 1234 ## Why SIGKILL cannot be ignored: ## SIGTERM: kernel delivers to process -> process handles it -> process exits ## SIGKILL: kernel kills the process directly -> process never even sees it ## nginx specifically uses SIGHUP to reload its config without downtime kill -HUP $(cat /var/run/nginx.pid) ## nginx receives SIGHUP -> re-reads nginx.conf -> gracefully drains old workers -> starts new workers ## Send signal to all processes with a name pkill -TERM nginx killall -TERM nginx ## Ctrl+C = SIGINT to the foreground process group (more on process groups later) ## Ctrl+Z = SIGSTOP to the foreground process group ## Ctrl+\ = SIGQUIT (like SIGTERM but also dumps core) ``` > 💡 **Tip:** Always try SIGTERM first and wait 5-10 seconds. If the process is well-written, it will shut down cleanly - flushing buffers, closing database connections, finishing in-flight requests. SIGKILL skips all of that and is how you cause data corruption, incomplete database transactions, and half-written log files.
This is the concept behind the Hotstar incident from the introduction. File descriptors are one of the most misunderstood parts of Linux, and exhausting them causes some of the strangest production failures - because the application appears healthy but silently cannot accept new connections or open new files. ### The three-layer model that explains everything Most people think: "a file descriptor is a number that represents an open file." This is true but incomplete. There are actually three layers involved, and understanding all three is what makes `lsof` output make sense. Layer 1: File Descriptor Table (per-process) Each process has its own table of integers: 0, 1, 2, 3, 4, ... Each integer is a "file descriptor" (FD) The FD just points to an entry in Layer 2 Layer 2: Open File Description Table (kernel-wide) One global table for the entire kernel Each entry holds: - current file offset (position for read/write) - open flags (read-only? write-only? append?) - pointer to the inode (Layer 3) Layer 3: Inode Table (filesystem) The actual file metadata: size, permissions, timestamps The actual data blocks on disk Process A Kernel Open File Table Filesystem Inodes FD 0 (stdin) -----> entry 0 (offset=0) -------> /dev/pts/0 FD 1 (stdout) ----> entry 1 (offset=0) -------> /dev/pts/0 FD 2 (stderr) ----> entry 2 (offset=0) -------> /dev/pts/0 FD 3 (log file) --> entry 3 (offset=4096) ----> /var/log/app.log FD 4 (socket) ----> entry 4 (offset=0) -------> [socket] **Why the three-layer model matters - the dup() case:** When you duplicate a file descriptor (`dup()` syscall), both FDs point to the **same Open File Description**. This means they share the same file offset. Reading through FD 5 advances the position that FD 6 also sees. This is how shell redirection works: `2>&1` duplicates stderr to point at the same Open File Description as stdout. ```bash ## See all file descriptors for a running process ls -la /proc/<PID>/fd/ ## Each symlink is one FD pointing to what it represents ## Example output for a web server process (PID 2847): ls -la /proc/2847/fd/ ## lrwxrwxrwx 0 -> /dev/null ## lrwxrwxrwx 1 -> /dev/null ## lrwxrwxrwx 2 -> /var/log/nginx/error.log ## lrwxrwxrwx 3 -> socket:[22183] <- listening socket ## lrwxrwxrwx 4 -> socket:[22184] <- client connection 1 ## lrwxrwxrwx 5 -> socket:[22185] <- client connection 2 ## ... potentially thousands of socket entries for a busy server ## Count how many FDs a process has open right now ls /proc/2847/fd | wc -l ## lsof gives you the same info in a friendlier format lsof -p 2847 | head -20 lsof -p 2847 | wc -l ## total FD count ## See FD usage across ALL processes sorted by count ## This finds the process leaking file descriptors lsof 2>/dev/null | awk '{print $2}' | sort | uniq -c | sort -rn | head -10 ``` ### Ulimits - the ceiling that kills your service at the worst moment Every process has **ulimits** - resource limits set by the kernel. The most critical for web services is `RLIMIT_NOFILE` - the maximum number of open file descriptors per process. There are two kinds of limit: * **Soft limit** - the current enforced limit. The process will get EMFILE error when it hits this. * **Hard limit** - the ceiling. The process can raise its soft limit up to the hard limit, but cannot exceed it. Only root can raise the hard limit. ```bash ## Check your current shell's limits ulimit -n ## soft limit for open files ulimit -Hn ## hard limit for open files ulimit -a ## all limits at once ## Typical output on a default Ubuntu server: ## open files (-n) 1024 <- way too low for a production web server ## Check limits for a RUNNING process (different from your shell) cat /proc/2847/limits ## Output: ## Limit Soft Limit Hard Limit Units ## Max open files 65536 65536 files ## Max processes 32768 32768 processes ## Increase limit temporarily (only up to hard limit) ulimit -n 65536 ## Increase permanently for a specific service via systemd ## Edit /etc/systemd/system/myapp.service: ## [Service] ## LimitNOFILE=65536 ## Increase permanently system-wide for all users echo "* soft nofile 65536" >> /etc/security/limits.conf echo "* hard nofile 65536" >> /etc/security/limits.conf ## Requires logout/login to take effect ## The kernel-wide maximum (ceiling for all processes combined) cat /proc/sys/fs/file-max ## And current usage cat /proc/sys/fs/file-nr ## Output: 12288 0 1048576 ## Meaning: 12288 open, 0 free slots in kernel table, 1048576 maximum ``` > ⚠️ **Security:** Setting ulimits too high (`LimitNOFILE=unlimited`) can cause a single misbehaving process to exhaust the kernel-wide file table. Set a high but finite value like 65536 or 524288 for production services. Always set this in the systemd unit file, not just in `/etc/security/limits.conf` - systemd units do not inherit PAM limits by default.
Running out of memory is one of the most misdiagnosed production problems. People see low `free` memory and panic. They add swap or restart services unnecessarily. Understanding what the kernel actually does with RAM prevents this - and helps you catch the cases where memory really is the problem. ### Virtual memory - every process lives in its own universe Each process thinks it owns the entire memory address space of the machine. A process running on a 4GB server might address memory location `0x7fff000` just like any other process - but they are not actually accessing the same physical RAM. This is **virtual memory**. The kernel maintains a **page table** for each process - a translation map that converts virtual addresses (what the process sees) to physical addresses (where the data actually is in RAM). Process thinks it owns all 64-bit address space: 0x0000000000000000 -> 0xFFFFFFFFFFFFFFFF Kernel's page table translates: Process virtual 0x400000 -> Physical RAM frame 0x1A3000 (code) Process virtual 0x600000 -> Physical RAM frame 0x2B1000 (data) Process virtual 0x7fff000 -> Physical RAM frame 0x0C2000 (stack) Process virtual 0x800000 -> Swap on disk (not in RAM right now) Benefits of virtual memory: * Processes are fully isolated - process A cannot read process B's memory even at the same virtual address * More total memory than physical RAM (via swap) * Memory can be shared between processes (shared libraries mapped to different virtual addresses but same physical frames) * Copy-on-write after `fork()` - parent and child share physical pages until one writes, then the kernel creates a private copy ```bash ## See a process's virtual memory layout cat /proc/2847/maps ## Output shows: start-end address, permissions, offset, device, inode, name ## 7f8a2c000000-7f8a2c200000 rw-p 00000000 00:00 0 ## 7f8a2c200000-7f8a2c400000 r--p 00000000 08:01 131074 /lib/x86_64-linux-gnu/libc.so ## Summary of virtual memory usage for a process cat /proc/2847/status | grep -E "VmRSS|VmSize|VmSwap|VmPeak" ## VmPeak: 512000 kB <- peak virtual memory size ever used ## VmSize: 480000 kB <- current virtual memory size ## VmRSS: 120000 kB <- Resident Set Size (actually in RAM right now) ## VmSwap: 8000 kB <- how much is on swap disk ## VmRSS is the real number - what is actually consuming physical RAM ## VmSize includes mapped but not loaded pages - always larger than VmRSS ``` ### The page cache - why "free" memory is misleading The kernel is smart about RAM. Any time you read a file, the kernel copies it into RAM and keeps it there in the **page cache**. Next time you read the same file, it comes from RAM instead of disk - orders of magnitude faster. The kernel uses every byte of available RAM for this cache. This is why `free -h` shows almost no free memory on a healthy, busy Linux server - the kernel has cached everything it can. This is **intentional and good**. ```bash free -h ## total used free shared buff/cache available ## Mem: 15Gi 3.2Gi 200Mi 512Mi 12Gi 11Gi ## "free" = 200MB looks alarming but is meaningless ## "available" = 11GB is what actually matters ## available = free + page cache that can be evicted instantly if needed ## The available column is what you monitor in production alerts ## Alert when available drops below your safety threshold (e.g. < 10% of total) ## See current page cache details cat /proc/meminfo | grep -E "MemTotal|MemFree|MemAvailable|Cached|Buffers|SwapTotal|SwapFree" ## MemTotal: 16384000 kB ## MemFree: 204800 kB <- nearly zero but that is fine ## MemAvailable: 11534336 kB <- this is what matters ## Buffers: 524288 kB <- block device cache ## Cached: 12386304 kB <- file page cache ## Drop the page cache (useful for benchmarking - not for production) ## sync first to write dirty pages to disk, then drop sync echo 3 | sudo tee /proc/sys/vm/drop_caches ``` ### The OOM Killer - how the kernel chooses who dies When the system truly runs out of memory (not page cache - actual allocatable memory), the kernel activates the **OOM (Out of Memory) Killer**. It has to kill something to free memory. The question is: what does it kill? The kernel scores each process with an **oom_score** from 0 to 1000. Higher score = more likely to be killed. The score is calculated based on: * How much memory the process uses (main factor - bigger = higher score) * How long it has been running (longer = slightly lower score) * Whether it has been manually adjusted via `oom_score_adj` `oom_score_adj` is the knob you control. Range: -1000 to +1000. * `-1000` = never kill this process (use for critical system daemons) * `0` = default, no adjustment * `+1000` = kill this first ```bash ## Check the OOM score of any process cat /proc/2847/oom_score ## Output: 342 <- relatively high, will be killed before lower-scored processes ## Check the current adjustment for a process cat /proc/2847/oom_score_adj ## Output: 0 <- no manual adjustment ## Protect a critical process from OOM killer ## (run as root - cannot be undone by the process itself) echo -1000 > /proc/$(pgrep postgres)/oom_score_adj ## Make a process more likely to be killed (e.g. a batch job that should die first) echo 500 > /proc/$(pgrep batch-worker)/oom_score_adj ## When OOM kill happens, it is in the kernel log dmesg | grep -i "oom\|killed process\|out of memory" ## Output: ## [1234567.890] Out of memory: Kill process 4821 (java) score 742 or sacrifice child ## [1234567.891] Killed process 4821 (java) total-vm:4096000kB, anon-rss:3584000kB ## The "score 742" tells you exactly why that process was chosen ## anon-rss is the anonymous memory (heap) that gets freed when it dies ## journalctl also catches OOM kills journalctl -k | grep -i "oom\|killed process" ``` > 📌 **Remember:** When Kubernetes says a pod was `OOMKilled`, it does NOT mean the host ran out of memory. It means the pod exceeded its memory **limit** (the cgroup boundary). The kernel's OOM killer fired within the cgroup, not for the whole host. You can have an OOMKilled pod on a host with 10GB free RAM.
During a production incident, tools like `ps`, `top`, and `htop` may not be installed (inside a minimal container), may be broken (their own processes failing), or may be too slow (you need to script a tight monitoring loop). `/proc` and `/sys` are always there - they are the kernel exposing its own state as files. No tool required. ### The /proc filesystem - the kernel's window into itself `/proc` is a **virtual filesystem** - it has no data on disk. Every file and directory you see is generated on-demand by the kernel when you read it. Reading `/proc/meminfo` does not read a log file - it asks the kernel for the current memory state right now. ```bash ## /proc/<PID>/ - everything about one specific process ls /proc/2847/ ## cmdline - the command that started this process ## cwd - symlink to current working directory ## environ - environment variables as null-separated string ## exe - symlink to the binary being executed ## fd/ - directory of FD symlinks (as shown earlier) ## limits - resource limits (ulimits) ## maps - virtual memory layout ## net/ - network stats for this process ## oom_score - OOM kill score ## smaps - detailed memory map with RSS per region ## stat - process stats (CPU times, state, PPID) ## status - human-readable process status ## Read what command launched a process (handles spaces in args correctly) cat /proc/2847/cmdline | tr '\0' ' ' ## Output: /usr/bin/python3 /opt/razorpay/payment-service/server.py --port 8080 ## Read environment variables for a process cat /proc/2847/environ | tr '\0' '\n' | grep -E "PORT|DB_HOST|NODE_ENV" ## Get process state, CPU time, memory from the raw stat file ## Field 3 is state: R=running S=sleeping D=disk wait Z=zombie T=stopped cat /proc/2847/stat | awk '{print "State:", $3, "| PPID:", $4, "| User CPU (jiffies):", $14, "| Sys CPU:", $15}' ## The status file is more human-readable cat /proc/2847/status ## Name: python3 ## State: S (sleeping) ## Pid: 2847 ## PPid: 2846 ## Threads: 8 ## VmRSS: 120000 kB ## VmSize: 480000 kB ``` ```bash ## Global system stats in /proc (not per-process) ## CPU info cat /proc/cpuinfo | grep -E "processor|model name|cpu MHz" | head -12 ## Memory info (the real numbers, not free's interpretation) cat /proc/meminfo ## Load average: 1min, 5min, 15min, runnable procs, total procs, last PID cat /proc/loadavg ## Output: 2.45 1.87 1.23 3/342 4821 ## 2.45 = 1min load, 3 = processes running RIGHT NOW, 342 = total processes ## Currently open files kernel-wide cat /proc/sys/fs/file-nr ## Output: 14328 0 1048576 ## 14328 open, 0 free cache slots, 1048576 max ## Network connections and their states cat /proc/net/tcp ## Column 3 is local address (hex), column 4 is remote, column 5 is state ## State 0A = LISTEN, 01 = ESTABLISHED, 06 = TIME_WAIT ## Disk I/O stats per device (updated in real time) cat /proc/diskstats ## Fields: major minor device reads_completed reads_merged sectors_read ... ``` ### /sys - tuning the kernel at runtime `/sys` exposes kernel parameters that you can read AND write to tune behaviour without rebooting. Changes to `/sys` are lost on reboot - use `/etc/sysctl.conf` for permanent changes. ```bash ## /sys/block/<device>/ - storage device parameters ls /sys/block/sda/queue/ ## scheduler - I/O scheduler (mq-deadline, kyber, bfq, none) ## nr_requests - depth of the I/O request queue ## read_ahead_kb - how much to pre-read ahead ## Check and change the I/O scheduler for SSD optimization cat /sys/block/sda/queue/scheduler ## Output: [mq-deadline] kyber bfq none <- current is in brackets ## For SSDs and NVMe, 'none' or 'mq-deadline' is usually better echo mq-deadline > /sys/block/sda/queue/scheduler ## Increase read-ahead for large sequential reads (backup, ETL) echo 4096 > /sys/block/sda/queue/read_ahead_kb ## /sys/class/net/<interface>/ - network interface parameters cat /sys/class/net/eth0/speed ## interface speed in Mbps cat /sys/class/net/eth0/statistics/rx_bytes ## bytes received cat /sys/class/net/eth0/statistics/tx_bytes ## bytes sent ## Key kernel parameters via sysctl (reads/writes /proc/sys/) sysctl vm.swappiness ## how aggressively to use swap (0-100) sysctl net.core.somaxconn ## max listen() queue depth for TCP connections sysctl net.ipv4.tcp_tw_reuse ## reuse TIME_WAIT sockets (useful for high-traffic servers) ## Tune for a high-throughput API server sysctl -w net.core.somaxconn=65535 sysctl -w net.ipv4.tcp_tw_reuse=1 sysctl -w vm.swappiness=10 ## prefer RAM over swap for server workloads ```
It is 3 AM. Hotstar's streaming service is throwing 503s for 2 million concurrent users during an IPL match. Your on-cal...
Before a single process runs, the kernel has to get off the ground. Understanding boot is not academic - it is how you d...
You already know processes exist. This section explains how they are born, how they communicate, and how they die at the...
This is the concept behind the Hotstar incident from the introduction. File descriptors are one of the most misunderstoo...
Running out of memory is one of the most misdiagnosed production problems. People see low free memory and panic. They ad...
During a production incident, tools like ps, top, and htop may not be installed (inside a minimal container), may be bro...
When you run kubectl apply -f deployment.yaml and set resources.limits.memory: 512Mi, something real happens in the kern...
A container is not a separate kernel. It is not a VM. It is a regular Linux process - the same as any other process on t...
The Linux CPU scheduler makes thousands of decisions per second about which process runs on which CPU core. Most of the ...
Disk I/O problems are one of the hardest to diagnose because they manifest as everything else - high CPU iowait, slow ap...
Network problems that look like application problems - connections timing out, intermittent failures under load, high la...
This section explains something most engineers use every day but rarely think about: why pressing Ctrl+C kills your runn...
With all the tools and internals covered above, you now need a mental framework for using them systematically. Ad-hoc de...
When USE tells you CPU utilisation is high but you do not know WHY - which function, which system call, which code path ...
This lab takes you through five real SRE diagnostic scenarios using kernel-level tools. Run each section in order on a L...
Essential /proc paths Path What it contains When to use it /proc/<pid>/status Process state, memory, threads First stop ...
Treating high iowait as a disk problem is one of the most expensive diagnostic mistakes in SRE work. iowait means the CP...
Aligns directly with DevOps, Site Reliability (SRE), and Platform Engineering job descriptions.