DevOps NetworkDevOpsNetwork
Join Free
Bash and Linux

Open Files per Process

mediumProcess management

Problem statement

Look inside /proc/PID/fd to see which files a process has open, watch the list change as files are opened and closed, spot a leak, and see what happens when an open file is deleted. "Too many open files" errors and disks that will not free up both come down to file descriptors.

The script opens files in its own shell, so it can inspect itself through /proc/$$/fd ($$ is the shell's own PID). It works in a temporary folder that contains:

b.txt

TEXT
hello
  1. Open a.log for writing as descriptor 3, b.txt for reading as 4 and c.log for appending as 5, then list the open files.
  2. Close descriptor 4 and list again.
  3. Open 3 more files and never close them, then count.
  4. Delete a.log while descriptor 3 still has it open, and show what /proc says.

Expected output:

◈ DIAGRAM
== files this shell has open ==
fd 3 -> a.log
fd 4 -> b.txt
fd 5 -> c.log
== after closing fd 4 ==
fd 3 -> a.log
fd 5 -> c.log
== a leak: opened 3 more and never closed them ==
open files here: 5
== a.log deleted while fd 3 still has it open ==
fd 3 -> a.log (deleted)

Hints

Hint 1: exec 3> a.log opens a file on descriptor 3 for the rest of the shell's life; exec 4<&- closes descriptor 4.

Approach

Optimal: Read /proc/PID/fd

Covers: file descriptors, /proc/PID/fd, readlink, exec N> file, exec N<&-, exec {fd}> (bash 4.1+), ulimit -n, lsof -p, deleted but open files.

A file descriptor is a numbered handle. When a program opens a file, the kernel gives it a small number, the file descriptor (fd). The program reads and writes through that number. Every process starts with three:

%%{init: {"flowchart": {"padding": 18, "nodeSpacing": 30, "rankSpacing": 40, "htmlLabels": true}, "themeVariables": {"fontSize": "18px"}}}%% flowchart TB P{{"a process"}}:::purple P --> F0["fd 0
stdin"]:::gray P --> F1["fd 1
stdout"]:::gray P --> F2["fd 2
stderr"]:::gray P --> F3["fd 3
a.log"]:::blue P --> F5["fd 5
c.log"]:::blue classDef blue fill:#dbeafe,stroke:#2563eb,color:#1e3a8a,stroke-width:2px classDef yellow fill:#fef3c7,stroke:#d97706,color:#78350f,stroke-width:2px classDef green fill:#d1fae5,stroke:#059669,color:#064e3b,stroke-width:2px classDef red fill:#fee2e2,stroke:#dc2626,color:#7f1d1d,stroke-width:2px classDef purple fill:#ede9fe,stroke:#7c3aed,color:#4c1d95,stroke-width:2px classDef gray fill:#f3f4f6,stroke:#6b7280,color:#111827,stroke-width:2px linkStyle default stroke:#94a3b8,stroke-width:2px
fd Name Usually
0 stdin keyboard or a pipe
1 stdout screen or a pipe
2 stderr screen
3 and up files, sockets, pipes the program opened

Sockets count too, so a web server with 10,000 connections has more than 10,000 descriptors open.

/proc shows them. Linux exposes every process as a folder in /proc. In /proc/PID/fd, each open descriptor is a link named by its number, pointing to what it is connected to. ls -l /proc/1201/fd on a server shows everything process 1201 has open. You can read your own processes; root can read all of them.

Opening and closing from the shell. exec with a redirect and no command changes the current shell's own descriptors:

Command Does
exec 3> a.log open a.log for writing on fd 3 (truncates it)
exec 4< b.txt open b.txt for reading on fd 4
exec 5>> c.log open c.log for appending on fd 5
exec 4<&- close fd 4
exec {fd}> x.log pick a free number (10 or more) and store it in $fd

Leaks and limits. A program that opens files or connections and forgets to close them leaks descriptors. The count only grows, until it hits the limit and every new open fails with "Too many open files". ulimit -n shows the limit for your shell, often 1024; services have their own, set with LimitNOFILE= in systemd. Watching the count over time is how you spot a leak.

Deleted, but still open. Deleting a file only removes its name. If a process still has it open, the data stays on disk and /proc shows the link as a.log (deleted). The space comes back only when the descriptor is closed, which is the cause of the Deleted Files Still Using Space page.

%%{init: {"flowchart": {"padding": 18, "nodeSpacing": 30, "rankSpacing": 40, "htmlLabels": true}, "themeVariables": {"fontSize": "18px"}}}%% flowchart LR subgraph NAME["rm a.log"] direction TB N1["the name is removed"]:::yellow end subgraph DATA["fd 3 still open"] direction TB D1["data stays on disk
a.log (deleted)"]:::red --> D2["space returns
when fd 3 closes"]:::green end NAME ~~~ DATA classDef blue fill:#dbeafe,stroke:#2563eb,color:#1e3a8a,stroke-width:2px classDef yellow fill:#fef3c7,stroke:#d97706,color:#78350f,stroke-width:2px classDef green fill:#d1fae5,stroke:#059669,color:#064e3b,stroke-width:2px classDef red fill:#fee2e2,stroke:#dc2626,color:#7f1d1d,stroke-width:2px classDef purple fill:#ede9fe,stroke:#7c3aed,color:#4c1d95,stroke-width:2px classDef gray fill:#f3f4f6,stroke:#6b7280,color:#111827,stroke-width:2px linkStyle default stroke:#94a3b8,stroke-width:2px style NAME fill:transparent,stroke:#d97706,stroke-width:2px style DATA fill:transparent,stroke:#dc2626,stroke-width:2px

Walking through the code. The # Setup: lines only create the folder and b.txt, so skip past them.

  1. list_fds loops over /proc/$$/fd/*, reads each link, and prints only links to files in this folder, so the shell's own pipes and terminal do not clutter the output. sort -k2,2n orders by descriptor number.
  2. After exec 4<&-, fd 4 is gone.
  3. The loop opens three files with exec {fd}> and keeps none of the numbers, a small leak; the count goes to 5.
  4. After rm a.log, fd 3 still points at it, marked (deleted).

Edge cases. /proc exists only on Linux; on macOS use lsof -p PID. A process can close and reopen files between two looks, so take several samples before calling it a leak.

# Setup: work in a fresh temporary folder with one file to read
cd "$(mktemp -d)"
echo "hello" > b.txt

# Show this shell's open descriptors that point at files in this folder
list_fds() {
  for p in /proc/$$/fd/*; do
    t=$(readlink "$p")
    [[ $t == "$PWD"/* ]] && echo "fd ${p##*/} -> ${t#"$PWD"/}"
  done | sort -k2,2n
}

exec 3> a.log 4< b.txt 5>> c.log
echo "== files this shell has open =="
list_fds

exec 4<&-
echo "== after closing fd 4 =="
list_fds

for i in 1 2 3; do exec {fd}> "leak-$i.log"; done
echo "== a leak: opened 3 more and never closed them =="
echo "open files here: $(list_fds | wc -l)"

rm a.log
echo "== a.log deleted while fd 3 still has it open =="
list_fds | grep 'fd 3 '

Interview follow-ups

  • Get back the space of a deleted log without restarting the process.

    Find the descriptor that still holds it, for example fd 3 -> app.log (deleted) in /proc/1201/fd. Truncate the file through that link: : > /proc/1201/fd/3. The process keeps its descriptor, but the file's size drops to zero and the space is freed. Be careful: the process may keep writing at its old position, which can leave a sparse file, so a planned restart or log reopen (kill -HUP for many daemons) is the clean fix.

Frequently asked questions

Count the entries in every process's fd folder and sort: for p in /proc/[0-9]*; do echo "$(ls $p/fd 2>/dev/null | wc -l) ${p#/proc/} $(cat $p/comm 2>/dev/null)"; done | sort -rn | head. Run it with sudo to see every process. lsof | awk '{print $2}' | sort | uniq -c | sort -rn | head gives a similar answer if lsof is installed, though lsof also lists memory-mapped files, so its numbers are higher.

For a systemd service, add LimitNOFILE=65536 under [Service] in an override (sudo systemctl edit name), then restart it. For login sessions, the limits live in /etc/security/limits.conf. ulimit -n 4096 raises it only for the current shell and the programs it starts, and only up to the hard limit (ulimit -Hn). Check the running process's real limit with cat /proc/PID/limits.

No. A busy server opens more connections when traffic grows and closes them when it drops, so the count follows load. A leak looks different: the count keeps climbing even when traffic is flat or at night, and never comes back down. Graph it next to the request rate before deciding. Restarting the process resets the count, which hides a leak for a while but does not fix it.