Out of Inodes
Problem statement
Find a filesystem that is out of inodes even though it still has free space, then find the folder that holds the most files. The symptom is confusing: "No space left on device" while df -h says the disk is only 60% used. The cause is millions of tiny files, and it is a favourite interview question.
df-h.txt (from df -h)
Filesystem Size Used Avail Use% Mounted on/dev/nvme0n1p1 50G 29G 21G 58% //dev/nvme1n1 200G 176G 24G 88% /datadf-i.txt (from df -i)
Filesystem Inodes IUsed IFree IUse% Mounted on/dev/nvme0n1p1 3276800 3276800 0 100% //dev/nvme1n1 13107200 410000 12697200 4% /dataThe script also builds a small tree of many small files (sessions/, cache/, logs/).
- Show that
/looks fine by space but is out of inodes. - Count the files in each top-level folder of the tree and print the busiest first.
Expected output:
== space vs inodes, per mount ==/ space 58% inodes 100% <-- out of inodes/data space 88% inodes 4%== files per folder, busiest first == 300 sessions 120 cache 5 logsHints
df -h, inode use is field 5 of df -i.Approach
Optimal: df -i, then count files per folder
Covers: what an inode is, df -i, IUse%, "No space left on device" with free space, find -xdev -type f, cut -d/ -f2, counting per folder, awk with two files.
Every file needs an inode. A filesystem keeps two separate budgets: space for the file contents, and inodes, the small records that describe each file (owner, permissions, size, where the data is). Most filesystems, like ext4, create a fixed number of inodes when the disk is formatted. Every file, folder and link uses one, even an empty file.
21G free"]:::green end subgraph INODE["inode budget"] direction TB I1["100% used
0 free"]:::red end SPACE ~~~ INODE INODE --> ERR["No space left on device"]:::red classDef blue fill:#dbeafe,stroke:#2563eb,color:#1e3a8a,stroke-width:2px classDef yellow fill:#fef3c7,stroke:#d97706,color:#78350f,stroke-width:2px classDef green fill:#d1fae5,stroke:#059669,color:#064e3b,stroke-width:2px classDef red fill:#fee2e2,stroke:#dc2626,color:#7f1d1d,stroke-width:2px classDef purple fill:#ede9fe,stroke:#7c3aed,color:#4c1d95,stroke-width:2px classDef gray fill:#f3f4f6,stroke:#6b7280,color:#111827,stroke-width:2px linkStyle default stroke:#94a3b8,stroke-width:2px style SPACE fill:transparent,stroke:#059669,stroke-width:2px style INODE fill:transparent,stroke:#dc2626,stroke-width:2px
So a disk can run out in two ways. Big files use up space. Millions of tiny files use up inodes while most of the space is still free. Either way, creating a new file fails with the same message: "No space left on device".
df -i shows the inode budget. It has the same layout as df, but counts inodes:
| Column | Means |
|---|---|
| Inodes | how many exist in total |
| IUsed | how many are used, one per file or folder |
| IFree | how many are left |
| IUse% | the share used |
In the sample, / has 21G of free space but 0 free inodes, so it is full.
Finding the folder with the most files. du measures space, so it will not show the culprit. Count files instead: list every file, keep the first part of the path, and count. Common causes are PHP or app session files, mail queues, cache folders, and temp files from a job that never cleans up.
top folder only"]):::purple C --> U(["sort | uniq -c | sort -rn"]):::purple U --> R["300 sessions
120 cache
5 logs"]:::green classDef blue fill:#dbeafe,stroke:#2563eb,color:#1e3a8a,stroke-width:2px classDef yellow fill:#fef3c7,stroke:#d97706,color:#78350f,stroke-width:2px classDef green fill:#d1fae5,stroke:#059669,color:#064e3b,stroke-width:2px classDef red fill:#fee2e2,stroke:#dc2626,color:#7f1d1d,stroke-width:2px classDef purple fill:#ede9fe,stroke:#7c3aed,color:#4c1d95,stroke-width:2px classDef gray fill:#f3f4f6,stroke:#6b7280,color:#111827,stroke-width:2px linkStyle default stroke:#94a3b8,stroke-width:2px
Walking through the code. The # Setup: lines only save the two df outputs and build the sample tree, so skip past them.
- The first awk reads
df-h.txtfirst (whileNR == FNR, it is still the first file) and stores each mount's space use. Then, for each line ofdf-i.txt, it prints space use and inode use side by side, with a warning at 90% or more.FNRis the line number in the current file, soFNR > 1skips each header. find . -xdev -type flists files without leaving this filesystem.cut -d/ -f2turns./sessions/sess_12intosessions.sort | uniq -c | sort -rncounts and ranks.
Edge cases. Some filesystems, like XFS and Btrfs, create inodes as needed, so they rarely run out. On a real server, start the count at the full mount, sudo find / -xdev -type f | cut -d/ -f2 | sort | uniq -c | sort -rn | head, then repeat inside the winner, one level deeper each time.
# Setup: save sample df outputs and build a tree of many small files, in a fresh temporary folder
cd "$(mktemp -d)"
cat > df-h.txt << 'OUT'
Filesystem Size Used Avail Use% Mounted on
/dev/nvme0n1p1 50G 29G 21G 58% /
/dev/nvme1n1 200G 176G 24G 88% /data
OUT
cat > df-i.txt << 'OUT'
Filesystem Inodes IUsed IFree IUse% Mounted on
/dev/nvme0n1p1 3276800 3276800 0 100% /
/dev/nvme1n1 13107200 410000 12697200 4% /data
OUT
mkdir -p tree/sessions tree/cache tree/logs
for i in $(seq 1 300); do : > "tree/sessions/sess_$i"; done
for i in $(seq 1 120); do : > "tree/cache/c_$i"; done
for i in $(seq 1 5); do : > "tree/logs/app.$i.log"; done
echo "== space vs inodes, per mount =="
awk 'NR == FNR { if (FNR > 1) space[$6] = $5; next }
FNR > 1 { warn = ($5 + 0 >= 90) ? " <-- out of inodes" : ""
printf "%-6s space %4s inodes %4s%s\n", $6, space[$6], $5, warn }' df-h.txt df-i.txt
echo "== files per folder, busiest first =="
cd tree
find . -xdev -type f | cut -d/ -f2 | sort | uniq -c | sort -rnRecapThe whole problem in a few lines, for the night before
- Spot it: "No space left on device" but
df -hshows free space - Idea:
df -ifor inode use, thenfind -xdev -type f | cut -d/ -f2 | sort | uniq -c | sort -rn - Cost: one walk of the tree; one line per file through the pipe
- Trap: only checking
df -h, or usingdu, which measures space, not file count
Interview follow-ups
Count files per folder one level deeper, inside the winner, without listing all of them by name.
Run the same pipeline from inside that folder, or keep the first two path parts:
find ./sessions -xdev -type f | cut -d/ -f2-3 | sort | uniq -c | sort -rn | head. The list of names still streams through the pipe, but nothing is stored except one counter per folder inuniq. On huge trees,LC_ALL=C sortspeeds up the sort a lot. Repeat level by level until you reach the folder that really grows.
Frequently asked questions
Delete files, because only removing files frees inodes; adding space does not. Find the folder with the most files as on this page, then clean it, for example find /var/lib/php/sessions -type f -mmin +1440 -delete for day-old sessions. If the folder is so big that rm * fails with "Argument list too long", find ... -delete still works. Then fix the cause, such as a job that never cleans up after itself.
Not on ext4: the inode count is set when the filesystem is created. You can choose more with mkfs.ext4 -i 4096 (one inode per 4 KiB) when formatting a new disk for many small files, or -N for an exact number. XFS and Btrfs allocate inodes as they need them, so they suit workloads with huge numbers of small files. For an existing ext4 disk, the options are cleaning up, moving the small files elsewhere, or reformatting.
Plain ls reads every name, then sorts them before printing anything, which takes a long time and a lot of memory. ls -f skips sorting and starts printing straight away. find dir -maxdepth 1 | head shows a sample without reading everything. Counting with find dir -maxdepth 1 -type f | wc -l is safe too. Spreading files across subfolders, like sessions/ab/abcd..., avoids huge single folders in the first place.