DevOps NetworkDevOpsNetwork
Join Free
Practical infra coding

Alert on a burst of 5xx responses in a sliding window

mediumLog parsing and aggregation

Problem statement

Each line of status.log is epoch_seconds status path, in time order. Raise an alert when there are 3 or more 5xx responses within the last 10 seconds, where the window at time t holds 5xx responses with timestamps greater than t - 10. Print the alert only once when it starts, and print a single RESOLVED line at the first line where the window drops below 3 again.

Print times as UTC HH:MM:SS. Skip lines that do not parse, and skip lines that go backwards in time, with a warning on stderr for each.

status.log

TEXT
1789898400 200 /api/users
1789898401 502 /api/orders
1789898403 200 /api/users
1789898404 503 /api/orders
1789898409 500 /api/orders
1789898410 200 /api/users
1789898412 502 /api/orders
1789898415 200 /api/users
1789898430 500 /api/orders
1789898431 500 /api/orders
1789898433 504 /api/orders

Examples

Example 1

Input: python solution.py status.log

Output: ALERT 10:00:09 3 x 5xx in the last 10s RESOLVED 10:00:15 ALERT 10:00:33 3 x 5xx in the last 10s

Explanation: At 10:00:09 the window holds 5xx at :01, :04 and :09. At :12 it holds :04, :09 and :12, which is still 3, so no second alert. At :15 the :04 entry has expired, leaving 2, so the alert resolves.

Hints

Approach

Optimal

A sliding window over time, implemented with a deque.

  1. Parse ts and status. Skip lines that fail, and lines older than the previous one: the deque logic assumes time only moves forward.
  2. If the status is 5xx, append ts on the right.
  3. Evict from the left while ts - recent[0] >= 10. Evicting on every line, not just 5xx lines, is what lets the alert resolve during a run of 200s.
  4. Compare len(recent) with the threshold and the alerting flag. Emit ALERT on the rising edge and RESOLVED on the falling edge.

Each timestamp is appended once and popped once, so the whole run is O(n). Memory is bounded by the number of 5xx responses inside one window (w). detect is a generator, so alerts are produced while the file is being read.

ComplexityTime O(n)Space O(w)
Python
import sys
from collections import deque
from datetime import datetime, timezone
WINDOW_S = 10
THRESHOLD = 3
def fmt(ts):
return datetime.fromtimestamp(ts, timezone.utc).strftime("%H:%M:%S")
def detect(lines, window=WINDOW_S, threshold=THRESHOLD):
recent = deque() # timestamps of 5xx responses inside the window, oldest first
alerting = False
last_ts = None
for lineno, line in enumerate(lines, 1):
parts = line.split()
if not parts:
continue
try:
ts, status = int(parts[0]), int(parts[1])
except (IndexError, ValueError):
print(f"line {lineno}: unparseable, skipped", file=sys.stderr)
continue
if last_ts is not None and ts < last_ts:
# the deque relies on time only moving forward
print(f"line {lineno}: out of order, skipped", file=sys.stderr)
continue
last_ts = ts
if 500 <= status <= 599:
recent.append(ts)
while recent and ts - recent[0] >= window:
recent.popleft()
if not alerting and len(recent) >= threshold:
alerting = True
yield f"ALERT {fmt(ts)} {len(recent)} x 5xx in the last {window}s"
elif alerting and len(recent) < threshold:
alerting = False
yield f"RESOLVED {fmt(ts)}"
def main(path):
with open(path, encoding="utf-8", errors="replace") as f:
for event in detect(f):
print(event)
if __name__ == "__main__":
main(sys.argv[1] if len(sys.argv) > 1 else "status.log")

Follow-up questions

  • Alert on the error rate (5xx / total > 20%) instead of a count. What do you keep in the window now?
  • Track windows per path, so one broken endpoint does not hide behind healthy ones.
  • Lines arrive up to 2 seconds late from different hosts. How do you handle that without false alerts?

Frequently asked questions

Fixed buckets miss bursts that straddle a boundary. Two errors at :08 and :09 and one at :11 are 3 errors within 3 seconds, but they fall into two buckets with counts of 2 and 1. A sliding window has no boundaries to miss.

It is a small version of what an alerting rule does, and the edge-triggering part is how alerting systems avoid paging someone once per log line. Interviewers look for amortised O(1) eviction and for correct window-boundary semantics.

Inserting in the middle of a deque is O(w) and complicates the alert state, because an alert you already printed could become wrong. Real pipelines accept a small lateness allowance by buffering a few seconds and sorting. Skipping with a warning is the honest simple version; say so in the interview.