DevOps NetworkDevOpsNetwork
Join Free
Practical infra coding

Run commands in parallel with per-command and overall timeouts

hardConcurrency basics

Problem statement

A small CI runner executes a set of shell-free commands (lint, tests, build...) and must never hang: every command has its own timeout, and the whole run has a hard deadline of 2.5 s. Write it with subprocess and a thread pool.

  • Run at most 2 commands at a time.
  • Each command gets min(its own timeout, time left before the deadline). A command that runs over is killed.
  • A command that cannot start before the deadline is reported as skipped, not started late.
  • Report one line per command, sorted by name, with a status and a detail:
    • OK and the last line of stdout;
    • FAIL with the exit code and the last line of stderr (or stdout);
    • TIMEOUT, saying whether its own timeout or the overall deadline stopped it;
    • ERROR if the program cannot be started at all;
    • SKIP if the deadline passed before it started.
  • Print ok/total and exit 1 unless everything is OK.

The jobs, in submission order (the fake commands use the current Python interpreter so they run anywhere):

Name Command Timeout
lint prints lint ok 5 s
unit exits 1 with 2 tests failed on stderr 5 s
slow-docs sleeps 10 s 1 s
integration sleeps 10 s 5 s
deploy-dry no-such-binary-xyz --dry-run 5 s
build prints built 12 files 5 s

Examples

Example 1

Input: python runner.py; echo "exit: $?"

Output: OK build built 12 files ERROR deploy-dry cannot start 'no-such-binary-xyz' TIMEOUT integration stopped at the overall deadline OK lint lint ok TIMEOUT slow-docs killed after 1s FAIL unit exit 1: 2 tests failed 2/6 ok exit: 1

Explanation: integration would be allowed 5 s, but it starts a moment after the run begins (once lint and unit finish), so its limit becomes whatever is left of the 2.5 s deadline. sys.exit('message') prints the message to stderr and exits with status 1.

Hints

Approach

Optimal

  1. Threads drive processes. Each job is a subprocess.run call in a pool thread. The thread only waits while the child runs, so threads are the right tool even though the GIL is present; max_workers=2 is the parallelism limit.
  2. One absolute deadline. The main thread computes deadline once with time.monotonic() (which, unlike wall-clock time, never jumps when the system clock is adjusted) and passes it to every job. When a job actually starts, it computes remaining. If nothing is left it returns SKIP without launching anything; otherwise its limit is min(own timeout, remaining). That is how a queued job can never push the run past the deadline.
  3. Timeouts kill. On TimeoutExpired, subprocess.run has already killed the child and collected it, so no zombie is left behind. Comparing limit with the job's own timeout tells you which rule stopped it, which is what someone reading a CI log needs to know.
  4. Every outcome is a result, not a crash. OSError (typically FileNotFoundError) means the program does not exist: ERROR. A non-zero exit is FAIL with the last useful line of stderr, falling back to stdout. Anything unexpected inside run_job is caught when reading the future, so one broken job cannot hide the others.
  5. Stable report. Results are keyed by name and printed sorted, whatever order they finished in. The exit code is 1 unless every job is OK.

j is the number of jobs. The run takes about 2.5 s here: the deadline, which cut integration short.

ComplexityTime O(j)Space O(j + output)
Python
import subprocess
import sys
import time
from concurrent.futures import ThreadPoolExecutor
PY = sys.executable
# name -> (command, per-command timeout in seconds)
JOBS = {
"lint": ([PY, "-c", "print('lint ok')"], 5),
"unit": ([PY, "-c", "import sys; sys.exit('2 tests failed')"], 5),
"slow-docs": ([PY, "-c", "import time; time.sleep(10)"], 1),
"integration": ([PY, "-c", "import time; time.sleep(10)"], 5),
"deploy-dry": (["no-such-binary-xyz", "--dry-run"], 5),
"build": ([PY, "-c", "print('built 12 files')"], 5),
}
MAX_PARALLEL = 2
OVERALL_DEADLINE = 2.5
def last_line(text):
lines = [l for l in (text or "").strip().splitlines() if l.strip()]
return lines[-1] if lines else ""
def run_job(name, cmd, timeout, deadline):
remaining = deadline - time.monotonic()
if remaining <= 0:
return "SKIP", "overall deadline passed before it started"
limit = min(timeout, remaining)
try:
r = subprocess.run(cmd, capture_output=True, text=True, timeout=limit)
except subprocess.TimeoutExpired: # run() kills the child for us
if limit < timeout:
return "TIMEOUT", "stopped at the overall deadline"
return "TIMEOUT", f"killed after {timeout}s"
except OSError:
return "ERROR", f"cannot start {cmd[0]!r}"
if r.returncode != 0:
return "FAIL", f"exit {r.returncode}: {last_line(r.stderr) or last_line(r.stdout)}"
return "OK", last_line(r.stdout)
def main():
deadline = time.monotonic() + OVERALL_DEADLINE
with ThreadPoolExecutor(max_workers=MAX_PARALLEL) as pool:
futures = {name: pool.submit(run_job, name, cmd, t, deadline)
for name, (cmd, t) in JOBS.items()}
results = {}
for name, fut in futures.items():
try:
results[name] = fut.result()
except Exception as e: # bug in run_job: report, keep going
results[name] = ("ERROR", repr(e))
for name in sorted(results):
status, detail = results[name]
print(f"{status:<8}{name:<12} {detail}")
bad = [n for n, (s, _) in results.items() if s != "OK"]
print(f"{len(results) - len(bad)}/{len(results)} ok")
return 1 if bad else 0
if __name__ == "__main__":
sys.exit(main())

Follow-up questions

  • Stream each command's output live with a [name] prefix instead of capturing it.
  • Add dependencies: build must only start after lint and unit succeed.
  • Retry a job once if it timed out on its own limit, but never past the overall deadline.

Frequently asked questions

CI runners, deploy orchestrators and node agents all launch external commands, and a classic operational failure is a job that hangs forever and blocks everything behind it. Interviewers look for per-command timeouts, an overall budget, clean handling of commands that cannot start, and a report that does not depend on timing.

Not always. subprocess.run kills the direct child. A shell script that started background processes can leave them running. On POSIX, start the job with start_new_session=True and kill the whole process group with os.killpg; on Windows, use a job object or taskkill /T.

It works well too, with asyncio.wait_for for timeouts and a Semaphore for the parallelism limit, and it avoids threads entirely. A thread pool with subprocess.run is shorter to write in an interview and easy to reason about; both designs need the same deadline logic.