DevOps NetworkDevOpsNetwork
Join Free
Practical infra coding

Merge time-sorted logs from several hosts

hardLog parsing and aggregation

Problem statement

Three web servers each wrote a log sorted by their own timestamps. You need a single timeline across all hosts to see what happened during an incident. The catch: the hosts are configured with different time zones, so the raw strings are not comparable. web2 writes +05:30 and web3 writes -04:00.

Merge the files into one stream in true chronological order, printing each line as UTC-timestamp host rest-of-line, where the host is the file name without .log. When timestamps are equal, keep the order in which the files were given. Lines without a valid timestamp are skipped with a warning.

Each real file is several GB, so you cannot load them into memory or concatenate and sort.

web1.log

TEXT
2026-09-20T10:00:01+00:00 GET /api/users 200
2026-09-20T10:00:04+00:00 GET /api/orders 500
2026-09-20T10:00:09+00:00 GET /api/users 200

web2.log

TEXT
2026-09-20T15:30:02+05:30 POST /api/login 200
2026-09-20T15:30:04+05:30 GET /api/orders 502
2026-09-20T15:30:07+05:30 GET /api/users 200

web3.log

TEXT
2026-09-20T06:00:03-04:00 GET /api/users 200
not-a-timestamp stray output from a crashed worker
2026-09-20T06:00:08-04:00 GET /api/orders 200

Examples

Example 1

Input: python solution.py web1.log web2.log web3.log

Output: 2026-09-20T10:00:01Z web1 GET /api/users 200 2026-09-20T10:00:02Z web2 POST /api/login 200 2026-09-20T10:00:03Z web3 GET /api/users 200 2026-09-20T10:00:04Z web1 GET /api/orders 500 2026-09-20T10:00:04Z web2 GET /api/orders 502 2026-09-20T10:00:07Z web2 GET /api/users 200 2026-09-20T10:00:08Z web3 GET /api/orders 200 2026-09-20T10:00:09Z web1 GET /api/users 200

Explanation: 15:30:02+05:30 is 10:00:02Z, and 06:00:03-04:00 is 10:00:03Z. Sorting the raw strings would put all of web3 first. stderr shows web3.log:2: bad timestamp, skipped. The two 10:00:04 lines keep file order: web1, then web2.

Hints

Approach

Optimal

Turn each file into a lazy stream of (utc_time, host, message) and merge the streams.

  1. events() is a generator per file. It splits off the first field, parses it with fromisoformat, treats naive timestamps as UTC (a stated assumption), and converts to UTC. Bad timestamps are reported with file:line and skipped.
  2. It also checks that each file really is sorted. heapq.merge trusts its inputs; one out-of-order file silently produces an out-of-order result, so a warning is the least you owe the reader.
  3. ExitStack opens every file and closes all of them even if one fails.
  4. heapq.merge keeps a heap with one entry per file. Each output line costs one pop and one push: O(log k).

Memory is O(k) lines, whatever the file sizes. Output is written as it is produced, so you can pipe it into grep or less and see results immediately.

ComplexityTime O(n log k)Space O(k)
Python
import heapq
import sys
from contextlib import ExitStack
from datetime import datetime, timezone
from pathlib import Path
def events(path, f):
"""Yield (utc_datetime, host, message) from one host's log, lazily."""
host = Path(path).stem
last = None
for lineno, line in enumerate(f, 1):
stamp, _, message = line.rstrip("\n").partition(" ")
if not stamp:
continue
try:
ts = datetime.fromisoformat(stamp)
except ValueError:
print(f"{path}:{lineno}: bad timestamp, skipped", file=sys.stderr)
continue
if ts.tzinfo is None:
ts = ts.replace(tzinfo=timezone.utc) # assumption: naive stamps are UTC
ts = ts.astimezone(timezone.utc)
if last is not None and ts < last:
# heapq.merge trusts its inputs; an unsorted file breaks the output order
print(f"{path}:{lineno}: out of order ({ts:%H:%M:%S} after {last:%H:%M:%S})",
file=sys.stderr)
last = max(ts, last) if last else ts
yield ts, host, message
def merge_logs(paths, out=sys.stdout):
with ExitStack() as stack:
streams = [events(p, stack.enter_context(open(p, encoding="utf-8", errors="replace")))
for p in paths]
# k-way merge: holds one pending line per file, so memory is O(k)
for ts, host, message in heapq.merge(*streams, key=lambda e: e[0]):
out.write(f"{ts:%Y-%m-%dT%H:%M:%SZ} {host} {message}\n")
if __name__ == "__main__":
merge_logs(sys.argv[1:] or ["web1.log", "web2.log", "web3.log"])

Follow-up questions

  • Some lines are multi-line stack traces without timestamps. How do you keep them attached to their event during the merge?
  • There are 2,000 files and the OS limits open file descriptors to 1,024. What do you do? (Merge in rounds into temporary files.)
  • Add --since and --until flags that start and stop reading each file early.

Frequently asked questions

That needs all n lines in memory and O(n log n) time, and prints nothing until the end. The inputs are already sorted, and a merge uses that fact. It is the same idea as the merge step of merge sort and the same as merging sorted linked lists.

Building one timeline from many hosts is standard incident work, and mixed time zones are a classic trap. Some hosts use local time, some UTC, some containers inherit the node's zone. The question checks the algorithm (k-way merge), streaming, and whether you normalise time before comparing it.

The merge is only as correct as the clocks. With NTP drift, events can appear in an impossible order, such as a response before its request. The fix is operational (keep NTP healthy) or needs causal IDs; no amount of sorting repairs skewed clocks. It is worth saying so.