DevOps NetworkDevOpsNetwork
Join Free
Build it yourself

Health checker with rise and fall thresholds

mediumMetrics and health

Problem statement

A load balancer must not pull a backend out of rotation because of one dropped probe, and must not send traffic to a backend the moment it answers once after a crash. Build the health-check state machine that decides, using a fall threshold (consecutive failures to mark a target unhealthy) and a rise threshold (consecutive successes to mark it healthy).

API (Java: runChecks(Probe probe) where Probe is boolean check(String target) throws Exception)

◈ DIAGRAM
HealthChecker(fall=3, rise=2)
add(target: str) -> None
remove(target: str) -> bool
run_checks(probe) -> list[str] # state changes, e.g. "web-2: healthy -> unhealthy"
state(target: str) -> "healthy" | "unhealthy"
healthy() -> list[str] # healthy targets, sorted by name

Rules

  • A new target starts unhealthy, and needs rise consecutive passes before it receives traffic.
  • run_checks(probe) calls probe(target) once per target, in name order. A pass is the probe returning True. Returning False, or raising any exception (a timeout, a refused connection), is a failure. One target's exception must not stop the others from being checked.
  • A healthy target becomes unhealthy after fall consecutive failures. An unhealthy target becomes healthy after rise consecutive passes. A pass resets the failure streak, and a failure resets the pass streak.
  • run_checks returns only the targets whose state changed this round, formatted "<target>: <old> -> <new>", in name order.
  • add on an existing target leaves its state alone. fall < 1 or rise < 1 raises ValueError.

Examples use fall=2, rise=2 with targets web-1 and web-2. Each input line is one round, showing what the probe returns per target. Example 2 continues from example 1.

Examples

Example 1

Input: round 1: web-1 True, web-2 True round 2: web-1 True, web-2 True round 3: web-1 True, web-2 raises TimeoutError round 4: web-1 True, web-2 False hc.healthy()

Output: [] [web-1: unhealthy -> healthy, web-2: unhealthy -> healthy] [] [web-2: healthy -> unhealthy] [web-1]

Explanation: Both need two passes to join. The timeout counts as web-2's first failure and the False as its second, which crosses fall = 2.

Example 2

Input: round 5: web-1 True, web-2 True round 6: web-1 True, web-2 False round 7: web-1 True, web-2 True round 8: web-1 True, web-2 True hc.state("web-2")

Output: [] [] [] [web-2: unhealthy -> healthy] healthy

Explanation: The failure in round 6 resets web-2's pass streak, so it needs rounds 7 and 8 to rise again. A flapping backend stays out.

Hints

Approach

Optimal

Per target keep state, ok_streak and fail_streak.

  1. run_checks iterates targets in sorted order, so output is deterministic. It calls the probe inside try/except. Only an explicit True counts as a pass, so a probe that accidentally returns None does not mark a dead host healthy.
  2. _record(target, ok): a pass increments ok_streak and zeroes fail_streak. If the target is unhealthy and ok_streak >= rise, it becomes healthy. A failure does the mirror image with fall.
  3. Remember the state before recording, and append "name: old -> new" if it changed.

The hysteresis, meaning different thresholds for going down and coming up, is what stops flapping: a backend that alternates pass and fail never builds the streak it needs to rejoin.

ComplexityTime O(t log t) per round for t targets (sorting); O(1) per targetSpace O(t)
Python
class _Target:
__slots__ = ("state", "ok_streak", "fail_streak")
def __init__(self):
self.state = "unhealthy" # must prove itself before getting traffic
self.ok_streak = 0
self.fail_streak = 0
class HealthChecker:
def __init__(self, fall=3, rise=2):
if fall < 1 or rise < 1:
raise ValueError("fall and rise must be at least 1")
self.fall, self.rise = fall, rise
self.targets = {}
def add(self, target):
self.targets.setdefault(target, _Target())
def remove(self, target):
return self.targets.pop(target, None) is not None
def state(self, target):
return self.targets[target].state
def healthy(self):
return sorted(t for t, s in self.targets.items() if s.state == "healthy")
def _record(self, t, ok):
"""Update streaks; return the new state if it changed, else None."""
if ok:
t.ok_streak += 1
t.fail_streak = 0
if t.state == "unhealthy" and t.ok_streak >= self.rise:
t.state = "healthy"
return t.state
else:
t.fail_streak += 1
t.ok_streak = 0
if t.state == "healthy" and t.fail_streak >= self.fall:
t.state = "unhealthy"
return t.state
return None
def run_checks(self, probe):
"""Probe every target once; return transitions as 'target: old -> new'."""
changes = []
for name in sorted(self.targets):
try:
ok = probe(name) is True # only an explicit True is a pass
except Exception:
ok = False # timeouts and errors are failures
t = self.targets[name]
before = t.state
if self._record(t, ok):
changes.append(f"{name}: {before} -> {t.state}")
return changes

Follow-up questions

  • Run the probes in parallel with a per-probe timeout.
  • Treat a slow response (over 500 ms) as a failure even when it returns 200.
  • Never mark more than half of the targets unhealthy at once (panic mode). Why would a load balancer want that?

Frequently asked questions

Starting unhealthy is the safe default for a load balancer: a freshly started instance may still be warming caches or running migrations, so it must prove itself first. Some systems start new targets in an initial or unknown state and send no traffic until the first verdict. Starting healthy risks sending traffic to an instance that is not ready, or never will be.

In Go each round would probe targets concurrently: one goroutine per target, each using context.WithTimeout for the probe, with results collected over a channel and then applied to the state map under a mutex. Probing sequentially means one hanging host delays every other check. In production also add jitter to the probe interval, emit transitions as events or metrics rather than only logs, and alert if too large a fraction of targets goes unhealthy at once. That pattern usually means the checker or the network is broken, not every backend.

Health checks decide where traffic goes, and they sit behind load balancers, Kubernetes readiness probes and service discovery. The question tests state-machine thinking, handling of failure modes, and the idea of hysteresis, all of which come up again in on-call work.