DevOps NetworkDevOpsNetwork
Join Free
Practical infra coding

Rotate logs and keep the last five

mediumFiles and filesystem

Problem statement

Implement numbered log rotation, like logrotate with rotate 5:

  • app.log becomes app.log.1, app.log.1 becomes app.log.2, and so on.
  • At most 5 old generations are kept. Anything that would become .6 or higher is deleted, including leftovers from when the setting was larger.
  • A missing generation (a gap) is not an error; shift what exists.
  • A new empty app.log is created.
  • If app.log is missing or empty, do nothing.

Return the list of actions and support a dry run that only reports them.

Sample directory

TEXT
app.log "today"
app.log.1 "day-1"
app.log.2 "day-2"
app.log.4 "day-4" (app.log.3 is missing)
app.log.5 "day-5"
app.log.6 "day-6" (left over from an older rotate 6 setting)

Examples

Example 1

Input: `python solution.py` (the demo builds the tree above in a temporary directory)

Output: before: app.log='today' app.log.1='day-1' app.log.2='day-2' app.log.4='day-4' app.log.5='day-5' app.log.6='day-6' delete app.log.6 delete app.log.5 rename app.log.4 -> app.log.5 rename app.log.2 -> app.log.3 rename app.log.1 -> app.log.2 rename app.log -> app.log.1 create app.log after: app.log='' app.log.1='today' app.log.2='day-1' app.log.3='day-2' app.log.5='day-4'

Explanation: Renames go from the highest number down, so nothing is overwritten before it has moved. The gap moves from .3 to .4.

Hints

Approach

Optimal

Discover, then shift from the top down.

  1. Return early if the log is missing or empty (notifempty).
  2. List the directory once and extract generation numbers with re.escape(log.name) + r"\.(\d+)" and fullmatch, so app.log.bak and app.log.1.gz are left alone. Sort the numbers descending.
  3. For each generation n: if n >= keep, delete it; otherwise rename it to n + 1. Going from the highest number down means every destination is already free.
  4. Rename the live log to .1 and create a new empty file.
  5. Every step goes through do(), which records the action and skips the file operation in dry-run mode, so the report and the real run cannot diverge.

g is the number of generations.

ComplexityTime O(g log g)Space O(g)
Python
import os
import re
import sys
import tempfile
from pathlib import Path
def rotate(log_path, keep=5, dry_run=False):
"""app.log -> app.log.1 -> ... -> app.log.<keep>; older copies are deleted."""
log = Path(log_path)
if not log.exists() or log.stat().st_size == 0:
return ["nothing to rotate"] # like logrotate's notifempty
actions = []
def do(msg, fn, *args):
actions.append(msg)
if not dry_run:
fn(*args)
# find existing generations by number, not by string sort (.10 sorts before .9)
pattern = re.compile(re.escape(log.name) + r"\.(\d+)")
gens = sorted((int(m[1]) for p in log.parent.iterdir()
if (m := pattern.fullmatch(p.name))), reverse=True)
for n in gens: # highest first, so nothing is overwritten before it moves
src = log.with_name(f"{log.name}.{n}")
if n >= keep:
do(f"delete {src.name}", os.remove, src)
else:
dst = log.with_name(f"{log.name}.{n + 1}")
do(f"rename {src.name} -> {dst.name}", os.replace, src, dst)
do(f"rename {log.name} -> {log.name}.1", os.replace, log, log.with_name(f"{log.name}.1"))
do(f"create {log.name}", log.touch)
return actions
def build_sample(base):
contents = {"app.log": "today", "app.log.1": "day-1", "app.log.2": "day-2",
"app.log.4": "day-4", "app.log.5": "day-5", "app.log.6": "day-6"}
for name, text in contents.items():
Path(base, name).write_text(text)
def show(base):
files = sorted(Path(base).iterdir(), key=lambda p: (len(p.name), p.name))
return " ".join(f"{p.name}={p.read_text()!r}" for p in files)
if __name__ == "__main__":
if len(sys.argv) > 1:
for a in rotate(sys.argv[1], dry_run="--dry-run" in sys.argv):
print(a)
else:
with tempfile.TemporaryDirectory() as tmp:
build_sample(tmp)
print("before:", show(tmp))
for a in rotate(Path(tmp, "app.log")):
print(a)
print("after: ", show(tmp))

Follow-up questions

  • Compress rotated generations (app.log.2.gz) but keep app.log.1 uncompressed, as logrotate's delaycompress does.
  • Rotate by size: only rotate when app.log is larger than 100 MiB.
  • Implement copytruncate instead of rename + create. What can be lost?

Frequently asked questions

To app.log.1. On Unix, a rename does not affect open file descriptors, so the process keeps writing to the same inode under its new name. Rotation therefore needs a second step: tell the process to reopen its log (nginx uses USR1, many daemons use HUP), or use copytruncate, which copies the file and truncates it in place, at the risk of losing lines written between the copy and the truncate.

Log rotation is simple to describe and easy to get wrong: wrong rename order, lexicographic sorting, and the open-file-handle issue above. It is also a common "debug this script" exercise, because broken rotation fills disks. Interviewers check the order of operations and whether you know the reopen problem.

They can interleave renames and delete the wrong generation. Guard the rotation with a lock, such as fcntl.flock on a lock file on Linux, or make sure only one scheduler runs it. logrotate avoids this by keeping a state file and running from one cron entry.