DevOps NetworkDevOpsNetwork
Join Free
Bash and Linux

Deleted Files Still Using Space

hardDisk and storage

Problem statement

Find files that were deleted but are still held open by a running process, add up the space they hold, and free that space without restarting anything. This explains the puzzle "df says the disk is full but du finds nothing", which usually starts when someone deletes a huge log that an app is still writing to.

lsof.txt (from sudo lsof +L1: open files with no name left on disk)

Bash
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NLINK NODE NAME
java 1201 app 12w REG 259,1 3221225472 0 524301 /var/log/app/app.log (deleted)
nginx 812 root 5w REG 259,1 104857600 0 524400 /var/log/nginx/access.log.1 (deleted)
python3 1610 app 3r REG 259,1 1048576 0 524500 /tmp/cache.db (deleted)
  1. List each process holding deleted files, with the space in MiB, biggest first, and the total.
  2. Recreate the problem in this shell: open a 2 MiB file, delete it, and show the space is still held.
  3. Free it by truncating the file through /proc/$$/fd, and show the size is now 0.

Expected output:

Bash
== processes holding deleted files ==
java 1201 3072.0 MiB /var/log/app/app.log
nginx 812 100.0 MiB /var/log/nginx/access.log.1
python3 1610 1.0 MiB /tmp/cache.db
total held: 3173.0 MiB
== recreate it: open a 2 MiB file, then delete it ==
fd 3 now points at: big.log (deleted)
space still held: 2097152 bytes
== free it without closing: truncate through /proc ==
space still held: 0 bytes

Hints

Hint 1: In lsof output, SIZE/OFF is field 7 and the file name is field 10. Skip the header with NR > 1 and divide the size by 1048576 for MiB.

Approach

Optimal: lsof +L1 and truncate via /proc

Covers: why df and du disagree, link count, lsof +L1, lsof columns, /proc/PID/fd, stat -L, truncating with : >, kill -HUP, logrotate copytruncate.

Deleting a file only removes its name. A file's data lives in an inode. Folders hold names that point to inodes, and the link count says how many names there are. rm removes a name. The data is freed only when both are true: no names are left, and no process has the file open.

%%{init: {"flowchart": {"padding": 18, "nodeSpacing": 30, "rankSpacing": 40, "htmlLabels": true}, "themeVariables": {"fontSize": "18px"}}}%% flowchart TB R(["rm app.log"]):::red --> N["name removed
du cannot see it"]:::yellow N --> O{{"is a process
still holding it?"}}:::purple O --> K["yes: data stays
df still counts it"]:::red O --> F["no: space freed"]:::green classDef blue fill:#dbeafe,stroke:#2563eb,color:#1e3a8a,stroke-width:2px classDef yellow fill:#fef3c7,stroke:#d97706,color:#78350f,stroke-width:2px classDef green fill:#d1fae5,stroke:#059669,color:#064e3b,stroke-width:2px classDef red fill:#fee2e2,stroke:#dc2626,color:#7f1d1d,stroke-width:2px classDef purple fill:#ede9fe,stroke:#7c3aed,color:#4c1d95,stroke-width:2px classDef gray fill:#f3f4f6,stroke:#6b7280,color:#111827,stroke-width:2px linkStyle default stroke:#94a3b8,stroke-width:2px

So if a program still writes to app.log and someone runs rm app.log to save space, nothing is saved. The program keeps writing into a file nobody can see. du cannot find it, because it has no name, but df still counts every block.

lsof +L1 finds them. lsof lists open files. +L1 keeps only files whose link count is less than 1, which means deleted but still open:

Column Field Means
COMMAND 1 the program
PID 2 its process ID
FD 4 the descriptor, 12w is fd 12 open for writing
SIZE/OFF 7 size in bytes
NLINK 8 link count, 0 here
NODE 9 the inode number
NAME 10 the old path, marked (deleted)

Freeing the space, three ways.

%%{init: {"flowchart": {"padding": 18, "nodeSpacing": 30, "rankSpacing": 40, "htmlLabels": true}, "themeVariables": {"fontSize": "18px"}}}%% flowchart LR subgraph FIX["ways to free it"] direction TB A1["restart the program"]:::yellow ~~~ A2["kill -HUP to reopen logs"]:::blue ~~~ A3["truncate /proc/PID/fd/N"]:::green end classDef blue fill:#dbeafe,stroke:#2563eb,color:#1e3a8a,stroke-width:2px classDef yellow fill:#fef3c7,stroke:#d97706,color:#78350f,stroke-width:2px classDef green fill:#d1fae5,stroke:#059669,color:#064e3b,stroke-width:2px classDef red fill:#fee2e2,stroke:#dc2626,color:#7f1d1d,stroke-width:2px classDef purple fill:#ede9fe,stroke:#7c3aed,color:#4c1d95,stroke-width:2px classDef gray fill:#f3f4f6,stroke:#6b7280,color:#111827,stroke-width:2px linkStyle default stroke:#94a3b8,stroke-width:2px style FIX fill:transparent,stroke:#6b7280,stroke-width:2px
  1. Restart the program. It closes the file and the space is freed. Simple, but it may cause downtime.
  2. Ask it to reopen its logs. Many daemons reopen their log files on kill -HUP PID (nginx uses nginx -s reopen), which closes the old deleted one.
  3. Truncate through /proc. : > /proc/1201/fd/12 empties the file right away. The program keeps running and keeps its descriptor; the space is freed now.

Walking through the code. The # Setup: lines only save the sample lsof output, so skip past them.

  1. The awk prints command, PID, MiB and path for each line, adds up a total, and sort -k3,3nr puts the biggest first. The total line is printed separately afterwards.
  2. head -c 2M /dev/zero > big.log makes a real 2 MiB file, exec 3< big.log keeps it open on fd 3, and rm deletes the name. readlink shows (deleted), and stat -L -c %s shows the 2 MiB are still there.
  3. : > /proc/$$/fd/3 truncates it; stat now shows 0.

Edge cases. lsof needs sudo to see other users' processes. Truncating a file the program is writing at a high offset can leave a sparse file that looks big but uses little space. The real fix is to stop deleting live logs: use logrotate, whose copytruncate option does the safe version of step 3 for you.

# Setup: save sample lsof +L1 output in a fresh temporary folder (on a server: sudo lsof +L1)
cd "$(mktemp -d)"
cat > lsof.txt << 'OUT'
COMMAND   PID USER   FD   TYPE DEVICE   SIZE/OFF NLINK    NODE NAME
java     1201 app    12w   REG  259,1 3221225472     0  524301 /var/log/app/app.log (deleted)
nginx     812 root    5w   REG  259,1  104857600     0  524400 /var/log/nginx/access.log.1 (deleted)
python3  1610 app     3r   REG  259,1    1048576     0  524500 /tmp/cache.db (deleted)
OUT

echo "== processes holding deleted files =="
awk 'NR > 1 { printf "%-8s %-5s %8.1f MiB  %s\n", $1, $2, $7 / 1048576, $10 }' lsof.txt | sort -k3,3nr
awk 'NR > 1 { t += $7 } END { printf "total held: %.1f MiB\n", t / 1048576 }' lsof.txt

echo "== recreate it: open a 2 MiB file, then delete it =="
head -c 2M /dev/zero > big.log
exec 3< big.log
rm big.log
echo "fd 3 now points at: $(basename "$(readlink /proc/$$/fd/3)")"
echo "space still held: $(stat -L -c %s /proc/$$/fd/3) bytes"

echo "== free it without closing: truncate through /proc =="
: > /proc/$$/fd/3
echo "space still held: $(stat -L -c %s /proc/$$/fd/3) bytes"
exec 3<&-

Interview follow-ups

  • Free the space held by every deleted log on the server in one go, carefully.

    List the links first: sudo find /proc/[0-9]*/fd -lname '/var/log/*(deleted)'. Look at each one and the process that owns it before touching anything, because a deleted database file is very different from a deleted log. Then truncate only the log ones: for fd in $(sudo find /proc/[0-9]*/fd -lname '/var/log/*(deleted)'); do sudo truncate -s 0 "$fd"; done. Afterwards, df -h should show the space back, and lsof +L1 should show those sizes as 0.

Frequently asked questions

Every deleted but open file shows up as a link ending in (deleted) under /proc/*/fd. sudo find /proc/[0-9]*/fd -lname '*(deleted)' -printf '%p -> %l\n' 2>/dev/null lists them all. For each one, stat -L -c %s on the link gives the size. This works on minimal servers and containers where lsof is not installed.

Because the program writing the log still has it open, so deleting it frees nothing until the program restarts, and you lose the log's contents too. To empty a live log safely, truncate it instead: : > /var/log/app/app.log or truncate -s 0 app.log, which keeps the same file and frees the space at once. Better still, set up logrotate so logs never grow that big.

copytruncate copies the live log to app.log.1, then truncates the original to zero. The program never notices, so it works with apps that cannot reopen their logs. The catch is a tiny window: lines written between the copy and the truncate are lost. Apps that support it should use the other method, rename the log and send a signal (postrotate with kill -HUP), which loses nothing.