DevOps NetworkDevOpsNetwork
Join Free
Practical infra coding

Stream a huge log: top clients by requests

mediumLog parsing and aggregation Must-do

Problem statement

The same question as a small log, but the grader runs your script on a 40 GB access log, some of it gzip-compressed from rotation, with a 512 MB memory limit. Report the three client IPs that made the most requests, with the total bytes sent to each. Break ties by bytes, larger first.

Your script must accept a plain access.log and a rotated access.log.1.gz without code changes. Memory may grow with the number of distinct IPs but not with the number of lines. Skip malformed lines and count them.

access.log

Bash
203.0.113.5 - - [20/Sep/2026:10:00:01 +0000] "GET /api/users HTTP/1.1" 200 512 "-" "curl/8.5.0"
198.51.100.7 - - [20/Sep/2026:10:00:02 +0000] "GET /static/app.js HTTP/1.1" 200 90000 "-" "Mozilla/5.0"
203.0.113.5 - - [20/Sep/2026:10:00:03 +0000] "POST /api/login HTTP/1.1" 200 128 "-" "curl/8.5.0"
192.0.2.44 - - [20/Sep/2026:10:00:05 +0000] "GET /api/users HTTP/1.1" 200 512 "-" "Mozilla/5.0"
203.0.113.5 - - [20/Sep/2026:10:00:06 +0000] "GET /api/orders HTTP/1.1" 200 2048 "-" "curl/8.5.0"
198.51.100.7 - - [20/Sep/2026:10:00:07 +0000] "GET /favicon.ico HTTP/1.1" 404 - "-" "Mozilla/5.0"
10.0.0.3 - - [20/Sep/2026:10:00:08 +0000] "GET /healthz HTTP/1.1" 200 2 "-" "kube-probe/1.31"
192.0.2.44 - - [20/Sep/2026:10:00:09 +0000] "GET /api/users?id=7 HTTP/1.1" 200 256 "-" "Mozilla/5.0"
203.0.113.5 - - [20/Sep/2026:10:00:11 +0000] "GET /api/orders HTTP/1.1" 200 2048 "-" "curl/8.5.0"
10.0.0.3 - - [20/Sep/2026:10:00:13 +0000] "GET /healthz HTTP/1.1" 200 2 "-" "kube-probe/1.31"
198.51.100.7 - - [20/Sep/2026:10:00:14 +0000] "GET /static/app.css HTTP/1.1" 200 30000 "-" "Mozilla/5.0"
10.0.0.3 - - [20/Sep/2026:10:00:18 +0000] "GET /healthz HTTP/1.1" 200 2 "-" "kube-probe/1.31"
192.0.2.44 - - [20/Sep/2026:10:00:19 +0000] "GET /api/us

Examples

Example 1

Input: python solution.py access.log

Output: 203.0.113.5 requests=4 bytes=4736 198.51.100.7 requests=3 bytes=120000 10.0.0.3 requests=3 bytes=6 skipped 1 malformed line(s)

Explanation: 198.51.100.7 and 10.0.0.3 tie on 3 requests; bytes break the tie. A bytes field of - counts as 0. The last line is truncated.

Example 2

Input: `python solution.py access.log.1.gz` (the same file, gzipped)

Output: 203.0.113.5 requests=4 bytes=4736 198.51.100.7 requests=3 bytes=120000 10.0.0.3 requests=3 bytes=6 skipped 1 malformed line(s)

Hints

Approach

Keep only aggregates in memory.

  1. open_log picks gzip.open or open from the file extension. Both return text iterators, so the rest of the code does not care which it got.
  2. for line in f holds one line at a time. Split on whitespace; a valid combined line has at least 10 fields with a numeric status in field 8 and a numeric or - byte count in field 9.
  3. Two Counters keyed by IP hold request counts and bytes.
  4. heapq.nlargest(k, ...) keeps a heap of size k while scanning the u distinct IPs.

Memory is O(u), the distinct client count, typically a few hundred thousand entries (tens of MB) even for a huge file. split() is also several times faster than a regex per line, which matters at this size.

ComplexityTime O(n + u log k)Space O(u)
Python
import gzip
import heapq
import sys
from collections import Counter
def open_log(path):
"""Text-mode handle for plain or gzip-compressed (rotated) logs."""
if path.endswith(".gz"):
return gzip.open(path, "rt", encoding="utf-8", errors="replace")
return open(path, encoding="utf-8", errors="replace")
def top_clients(path, n=3):
requests = Counter()
sent = Counter()
bad = 0
with open_log(path) as f:
for line in f: # the file object is a lazy iterator: one line in memory at a time
parts = line.split()
# combined format: ip - user [date tz] "method path proto" status bytes ...
if len(parts) < 10 or not parts[8].isdigit():
bad += 1
continue
size = parts[9]
if size != "-" and not size.isdigit():
bad += 1
continue
ip = parts[0]
requests[ip] += 1
sent[ip] += 0 if size == "-" else int(size)
# nlargest keeps a heap of size n instead of sorting every unique IP
top = heapq.nlargest(n, requests.items(), key=lambda kv: (kv[1], sent[kv[0]]))
return [(ip, count, sent[ip]) for ip, count in top], bad
def main(path):
top, bad = top_clients(path)
for ip, count, size in top:
print(f"{ip} requests={count} bytes={size}")
print(f"skipped {bad} malformed line(s)")
if __name__ == "__main__":
main(sys.argv[1] if len(sys.argv) > 1 else "access.log")

Follow-up questions

  • Process all rotated files access.log, access.log.1, access.log.2.gz ... in one run and combine the counts.
  • The file is still being written. How do you process only the lines added since your last run? (Store the byte offset and inode; detect rotation when the inode changes or the file shrinks.)
  • How would you split the work across 8 CPU cores? (Split by byte ranges aligned to newlines, count per chunk with multiprocessing, merge the Counters.)

Frequently asked questions

Partition first: stream the file once and append each line to one of, say, 64 bucket files chosen by hash(ip) % 64. Each IP lands in exactly one bucket, so you can count each bucket separately and keep a running top 3. That is external aggregation. If an approximate answer is acceptable, a Count-Min sketch uses fixed memory.

Interviewers often start with a small file and then say "now it is 50 GB". They want to see that you already streamed the input, and that you can explain where memory goes. It separates people who have processed production logs from people who have only processed sample files.

Up to the bytes field, yes: IP, identity, user, the two halves of [date tz], the three words of the request, status and bytes are space-separated and never quoted in a way that adds spaces. The user agent comes later, so its spaces do not matter. If the request line has an unusual number of words, the status check on field 8 fails and the line is counted as malformed.