DevOps NetworkDevOpsNetwork
Join Free
Practical infra coding

Total file size per extension under a directory

easyFiles and filesystem Must-do

Problem statement

A volume is filling up and you want to know what kind of files are using it. Walk a directory tree and report, for each file extension, how many files there are and their total size, largest total first. Break ties by extension name.

  • Compare extensions case-insensitively: .YAML and .yaml are the same.
  • Files with no extension, including dotfiles such as .env, go under (none).
  • Count only regular files. Do not follow symlinks, which could loop or point outside the tree.
  • A directory you are not allowed to read must produce a warning, not a crash.

Print sizes in binary units (KiB) with one decimal, and end with a total line.

Sample tree (size in bytes)

TEXT
project/
README.md 1500
Makefile 400
.env 120
src/main.py 4000
src/util.py 2500
src/config.YAML 800
deploy/app.yaml 1200
deploy/values.yaml 600
logs/app.log 10000
logs/app-2026-09-19.log.gz 3000

Examples

Example 1

Input: `python solution.py` (the demo builds the tree above in a temporary directory)

Output: .log 1 file(s) 9.8 KiB .py 2 file(s) 6.3 KiB .gz 1 file(s) 2.9 KiB .yaml 3 file(s) 2.5 KiB .md 1 file(s) 1.5 KiB (none) 2 file(s) 520 B total 10 file(s) 23.6 KiB

Explanation: .yaml is 800 + 1200 + 600 = 2600 bytes, which is 2.5 KiB. Makefile and .env have no extension. The rotated log counts as .gz, its last suffix.

Hints

Approach

Optimal

One walk, two dicts.

  1. os.walk(root, onerror=errors.append) visits every directory. Directories that cannot be listed are passed to onerror instead of raising, so they become warnings.
  2. For each file name, os.lstat it. A file can disappear between the listing and the stat, which is common on a busy log directory, so an OSError is recorded and skipped.
  3. Skip anything that is not a regular file: symlinks, sockets, FIFOs, device files.
  4. Add the size to sizes[ext] and one to counts[ext], with ext lower-cased or (none).
  5. Sort the e extensions by (-size, ext) and format with human().

f is the number of files. Memory is the number of distinct extensions plus the walk's directory stack (d); the tree itself is never held in memory.

ComplexityTime O(f + e log e)Space O(e + d)
Python
import os
import stat
import sys
import tempfile
from collections import defaultdict
from pathlib import Path
def human(n):
for unit in ("B", "KiB", "MiB", "GiB"):
if n < 1024 or unit == "GiB":
return f"{n} {unit}" if unit == "B" else f"{n:.1f} {unit}"
n /= 1024
def size_by_extension(root):
sizes = defaultdict(int)
counts = defaultdict(int)
errors = []
# os.walk does not descend into symlinked directories by default;
# onerror collects directories we are not allowed to list instead of crashing
for dirpath, _dirs, files in os.walk(root, onerror=errors.append):
for name in files:
path = os.path.join(dirpath, name)
try:
st = os.lstat(path) # lstat: do not follow symlinks
except OSError as e: # deleted between listing and stat, etc.
errors.append(e)
continue
if not stat.S_ISREG(st.st_mode):
continue # symlinks, sockets, FIFOs
ext = Path(name).suffix.lower() or "(none)"
sizes[ext] += st.st_size
counts[ext] += 1
return sizes, counts, errors
def report(root):
sizes, counts, errors = size_by_extension(root)
for ext in sorted(sizes, key=lambda e: (-sizes[e], e)):
print(f"{ext:<7} {counts[ext]:>2} file(s) {human(sizes[ext]):>9}")
print(f"{'total':<7} {sum(counts.values()):>2} file(s) {human(sum(sizes.values())):>9}")
for e in errors:
print(f"warning: {e}", file=sys.stderr)
SAMPLE = { # relative path -> size in bytes
"README.md": 1500, "Makefile": 400, ".env": 120,
"src/main.py": 4000, "src/util.py": 2500, "src/config.YAML": 800,
"deploy/app.yaml": 1200, "deploy/values.yaml": 600,
"logs/app.log": 10000, "logs/app-2026-09-19.log.gz": 3000,
}
def build_sample(base):
for rel, size in SAMPLE.items():
p = Path(base, rel)
p.parent.mkdir(parents=True, exist_ok=True)
p.write_bytes(b"x" * size)
if __name__ == "__main__":
if len(sys.argv) > 1:
if not os.path.isdir(sys.argv[1]):
sys.exit(f"not a directory: {sys.argv[1]}")
report(sys.argv[1])
else:
with tempfile.TemporaryDirectory() as tmp:
build_sample(tmp)
report(tmp)

Follow-up questions

  • Treat .tar.gz and .log.1 style names as one logical extension (.tar.gz, .log).
  • Only count files larger than 1 MiB, and list the 5 largest per extension.
  • The tree has 50 million files on network storage. How would you parallelise the walk? (A worker pool over top-level directories, with os.scandir to avoid extra stat calls.)

Frequently asked questions

getsize follows symlinks, so a link to a large file is counted twice and a link to / can make the walk explode. The glob version also stops with an exception at the first unreadable directory or vanished file. It works on a clean sample and fails on a real server.

This script adds up apparent sizes (st_size). df and du report allocated blocks, which differ for sparse files, for filesystem block rounding, and for files that are deleted but still held open by a process. Deleted-but-open log files are a classic reason why df says full while du finds nothing; lsof +L1 finds them.

"Disk is full, find out why" is a very common on-call page. The script checks that you can walk a filesystem safely: no symlink loops, no crash on permissions, no double counting. The same concerns appear in every backup and cleanup tool.