DevOps NetworkDevOpsNetwork
Join Free
Practical infra coding

Count status codes and top 3 paths in an nginx log

easyLog parsing and aggregation Must-do

Problem statement

You are handed an nginx access log in the default combined format and asked two questions: how many responses of each HTTP status code were served, and which three paths were requested most. This is the most common opening task of an infra coding round.

Rules:

  • Count a path without its query string: /api/orders?page=2 counts as /api/orders.
  • Break ties in the top 3 alphabetically by path.
  • Print status codes in ascending order.
  • The log was copied while it was being rotated, so one line is cut off. Do not crash on it; count how many lines you skipped.
  • When a client disconnects before sending a request, nginx logs the request as "-". Count its status, but it has no path.

access.log

Bash
203.0.113.5 - - [20/Sep/2026:10:00:01 +0000] "GET /api/users HTTP/1.1" 200 512 "-" "curl/8.5.0"
198.51.100.7 - - [20/Sep/2026:10:00:02 +0000] "GET /api/orders?page=2 HTTP/1.1" 200 2048 "-" "Mozilla/5.0"
203.0.113.5 - - [20/Sep/2026:10:00:03 +0000] "POST /api/login HTTP/1.1" 200 128 "-" "curl/8.5.0"
192.0.2.44 - - [20/Sep/2026:10:00:05 +0000] "GET /api/users HTTP/1.1" 200 512 "-" "Mozilla/5.0"
198.51.100.7 - - [20/Sep/2026:10:00:06 +0000] "GET /favicon.ico HTTP/1.1" 404 0 "-" "Mozilla/5.0"
203.0.113.9 - - [20/Sep/2026:10:00:08 +0000] "GET /api/orders HTTP/1.1" 500 97 "-" "python-requests/2.32"
192.0.2.44 - - [20/Sep/2026:10:00:09 +0000] "GET /api/us
192.0.2.44 - - [20/Sep/2026:10:00:11 +0000] "GET /api/users?id=7 HTTP/1.1" 200 256 "-" "Mozilla/5.0"
203.0.113.5 - - [20/Sep/2026:10:00:12 +0000] "GET /api/orders HTTP/1.1" 200 2048 "-" "curl/8.5.0"
198.51.100.7 - - [20/Sep/2026:10:00:14 +0000] "GET /wp-login.php HTTP/1.1" 404 0 "-" "sqlmap/1.8"
203.0.113.9 - - [20/Sep/2026:10:00:15 +0000] "GET /api/users HTTP/1.1" 304 0 "-" "Mozilla/5.0"

Write summarize(path) and a script that prints the report below.

Examples

Example 1

Input: python solution.py access.log

Output: Status codes: 200 6 304 1 404 2 500 1 Top 3 paths: /api/users 4 /api/orders 3 /api/login 1 Skipped 1 malformed line(s)

Explanation: Line 7 is truncated and is skipped. /api/users appears 4 times once ?id=7 is stripped. Three paths are tied at 1; /api/login comes first alphabetically.

Hints

Approach

Optimal

Stream the file one line at a time and match each line against a regex for the fixed-position part of the combined format: IP, two ignored fields, [time], the quoted request, status and bytes. The regex only needs to reach the bytes field; referrer and user agent are ignored.

  1. Skip blank lines. Count any line the regex does not match as malformed and move on.
  2. Add the status to one Counter.
  3. If the request had a target, cut it at the first ? and add it to a second Counter.
  4. Sort the path counter by (-count, path) and take the first three.

n is the number of lines, s the number of distinct status codes and p the number of distinct paths. Memory depends on how many distinct paths there are, not on file size. errors="replace" means a stray invalid byte cannot raise UnicodeDecodeError halfway through a large file.

ComplexityTime O(n + p log p)Space O(s + p)
Python
import re
import sys
from collections import Counter
# ip - user [time] "METHOD target PROTO" status bytes ...
# nginx writes the request as "-" when a client disconnects before sending one.
LINE_RE = re.compile(
r'^(?P<ip>\S+) \S+ \S+ \[(?P<time>[^\]]+)\] '
r'"(?:(?P<method>[A-Z]+) (?P<target>\S+)[^"]*|-)" '
r'(?P<status>\d{3}) (?P<bytes>\d+|-)'
)
def summarize(path, top_n=3):
statuses = Counter()
paths = Counter()
bad = 0
with open(path, encoding="utf-8", errors="replace") as f:
for line in f: # one line at a time, never the whole file
if not line.strip():
continue
m = LINE_RE.match(line)
if not m:
bad += 1
continue
statuses[m["status"]] += 1
if m["target"]:
paths[m["target"].split("?", 1)[0]] += 1 # drop the query string
top = sorted(paths.items(), key=lambda kv: (-kv[1], kv[0]))[:top_n]
return statuses, top, bad
def main(path):
statuses, top, bad = summarize(path)
print("Status codes:")
for code in sorted(statuses):
print(f" {code} {statuses[code]}")
print("Top 3 paths:")
for p, n in top:
print(f" {p} {n}")
print(f"Skipped {bad} malformed line(s)")
if __name__ == "__main__":
main(sys.argv[1] if len(sys.argv) > 1 else "access.log")

Follow-up questions

  • The log is 40 GB and gzipped. What changes? (Open it with gzip.open(path, "rt"); the loop stays the same.)
  • Only count requests from the last 15 minutes. How do you parse the [20/Sep/2026:10:00:01 +0000] timestamp, and can you stop reading early?
  • The paths contain IDs (/api/users/42). How would you group them into routes like /api/users/:id?

Frequently asked questions

split() works on well-formed lines, but a truncated line gives fewer fields and parts[8] raises IndexError. A request of "-" also moves every later field by one position. The anchored regex either matches the whole prefix or it does not, so you know which lines are malformed instead of silently counting the wrong field.

Access logs are the first thing you read during an incident: a spike of 5xx, a path being scraped, a bot hammering login. Interviewers use this task to check that you stream the input, handle bad lines, and produce deterministic output. Those same habits matter when you write the real script at 3 a.m.

most_common orders ties by insertion order, so the output depends on the order of lines in the file. With an explicit (-count, path) key, the same data always gives the same answer, which matters when a test compares output exactly.