DevOps NetworkDevOpsNetwork
Join Free
Practical infra coding

Group Java stack traces by root cause

mediumLog parsing and aggregation

Problem statement

A Java service logs one event per line, except that an ERROR event can be followed by a stack trace spanning many lines. A new event always starts with a YYYY-MM-DD HH:MM:SS LEVEL prefix; anything else belongs to the event before it.

For every ERROR event, find its root cause: the last exception in the chain (the last Caused by: line, or the first exception line if there is no Caused by:). An error with no trace counts as (no stack trace). Print each root cause with its count and the timestamp it first appeared. Most frequent first; ties by first appearance.

service.log

TEXT
2026-09-20 10:00:01 INFO OrderService - order 1001 created
2026-09-20 10:00:03 ERROR OrderService - failed to charge card
java.lang.IllegalStateException: payment gateway returned 502
at com.shop.pay.Gateway.charge(Gateway.java:88)
at com.shop.order.OrderService.checkout(OrderService.java:41)
Caused by: java.net.SocketTimeoutException: Read timed out
at java.base/java.net.SocketInputStream.read(SocketInputStream.java:168)
... 12 more
2026-09-20 10:00:04 INFO OrderService - order 1002 created
2026-09-20 10:00:09 ERROR InventoryService - stock lookup failed
java.lang.NullPointerException: sku is null
at com.shop.inv.Stock.find(Stock.java:23)
2026-09-20 10:00:15 ERROR OrderService - failed to charge card
java.lang.IllegalStateException: payment gateway returned 502
at com.shop.pay.Gateway.charge(Gateway.java:88)
Caused by: java.net.SocketTimeoutException: Read timed out
... 9 more
2026-09-20 10:00:20 ERROR ReportJob - nightly report skipped

Examples

Example 1

Input: python solution.py service.log

Output: 2 java.net.SocketTimeoutException first=2026-09-20 10:00:03 1 java.lang.NullPointerException first=2026-09-20 10:00:09 1 (no stack trace) first=2026-09-20 10:00:20

Explanation: Both payment failures are reported as IllegalStateException, but the real cause is the socket timeout underneath. Grouping by the surface exception would hide that.

Hints

Approach

Optimal

Split the problem in two: turn lines into records, then classify each record.

  1. records() is a generator with one piece of state, the record being built. When a line matches RECORD_START, it yields the previous record and starts a new one. Otherwise the line is appended. Lines before the first record (the file begins in the middle of a trace after rotation) are counted and ignored. A cap of 500 lines stops a runaway trace from using unbounded memory.
  2. root_cause() scans a record's continuation lines and keeps the last line that matches the exception pattern. Frames (at ...) and ... 12 more do not match.
  3. A Counter counts causes and setdefault stores the first timestamp.

Time is linear in lines. Memory is one record (r) plus one entry per distinct cause (c).

ComplexityTime O(n)Space O(r + c)
Python
import re
import sys
from collections import Counter
RECORD_START = re.compile(r"^(\d{4}-\d\d-\d\d \d\d:\d\d:\d\d) (\w+)\s")
# "java.net.SocketTimeoutException: Read timed out" or "Caused by: ..."
EXCEPTION = re.compile(r"^(?:Caused by: )?([A-Za-z_$][\w$]*(?:\.[\w$]+)+)(?::|$)")
MAX_LINES = 500 # stop a runaway trace from eating memory
def records(lines):
"""Yield (timestamp, level, [lines]) for each log event, trace lines included."""
current = None
orphans = 0
for raw in lines:
line = raw.rstrip("\n")
m = RECORD_START.match(line)
if m:
if current:
yield current
current = (m[1], m[2], [line])
elif current is None:
orphans += 1 # file starts mid-trace (e.g. just after rotation)
elif len(current[2]) < MAX_LINES:
current[2].append(line)
if current:
yield current
if orphans:
print(f"ignored {orphans} line(s) before the first record", file=sys.stderr)
def root_cause(record_lines):
"""The last exception in the chain is the one that started it."""
cause = None
for line in record_lines[1:]:
m = EXCEPTION.match(line)
if m:
cause = m[1]
return cause or "(no stack trace)"
def summarize(lines):
counts = Counter()
first_seen = {}
for ts, level, body in records(lines):
if level != "ERROR":
continue
cause = root_cause(body)
counts[cause] += 1
first_seen.setdefault(cause, ts)
return sorted(counts.items(), key=lambda kv: (-kv[1], first_seen[kv[0]])), first_seen
def main(path):
with open(path, encoding="utf-8", errors="replace") as f:
rows, first_seen = summarize(f)
for cause, n in rows:
print(f"{n} {cause} first={first_seen[cause]}")
if __name__ == "__main__":
main(sys.argv[1] if len(sys.argv) > 1 else "service.log")

Follow-up questions

  • Support Python tracebacks, where the exception is the last line of the block and there is no dot in ValueError: bad input.
  • Group by root cause plus the first application frame (com.shop...), so two different bugs that both raise NullPointerException are counted separately.
  • The log is a live stream. When is a record finished if no new event arrives? (Flush after a timeout.)

Frequently asked questions

Traces have variable length, so a fixed context size either cuts traces short or pulls in unrelated lines. You need to know where each record ends, which means recognising the start of the next one. That is the whole idea of multi-line log parsing, and it is what log shippers like Fluent Bit and Logstash configure with a multiline start pattern.

During incidents you often get "the service is throwing errors" and a large log. Grouping by root cause turns thousands of lines into two or three distinct problems. Configuring multi-line parsing correctly is also a real task when shipping JVM logs to a central store.

A message such as failed to load com.shop.Config: missing on its own continuation line could match. Here only continuation lines are checked, never the event's first line, which limits the risk. A stricter pattern could require the class name to end in Exception or Error, at the cost of missing custom Throwable names.