DevOps NetworkDevOpsNetwork
Join Free
Practical infra coding

Classify HTTP endpoints as up or down with urllib

easyAPIs and HTTP

Problem statement

Write check(url, timeout=0.5) that requests a URL with urllib.request.urlopen and returns a verdict and a short detail string. It must never raise for a network problem; a health checker that crashes on the first bad host is useless.

  • 2xx or 3xx that ends in a 2xx after redirects: UP with the final status. If the request was redirected, add where it ended up.
  • 4xx or 5xx: DOWN with the status code.
  • No answer within timeout: DOWN timeout.
  • Connection refused or DNS failure: DOWN unreachable.

To test offline, the script starts a local http.server on 127.0.0.1 with these endpoints:

Path Behaviour
/healthz 200 ok
/old-health 301 redirect to /healthz
/broken 500
anything else 404
/slow waits 1.5 s, then 200 (longer than the 0.5 s timeout)

It also checks a port on 127.0.0.1 where nothing is listening, with a 5 s timeout: Linux refuses such a connection at once, but Windows retries for about 2 s before reporting it refused, which a 0.5 s timeout would misreport as timeout. Print one aligned line per URL (verdict, detail, path) and a final count.

Examples

Example 1

Input: `python solution.py` (checks `/healthz`, `/old-health`, `/broken`, `/missing`, `/slow` and a closed port)

Output: UP 200 /healthz UP 200 (redirected to healthz) /old-health DOWN 500 /broken DOWN 404 /missing DOWN timeout /slow DOWN unreachable /closed-port/healthz 4 of 6 down

Explanation: urlopen follows the 301 by itself, so the response is the 200 from /healthz; comparing resp.url with the requested URL reveals the redirect.

Hints

Approach

Optimal

check wraps one urlopen call and maps every outcome to a verdict:

  1. Normal response. urlopen follows redirects automatically, so a 301 to a healthy page arrives as a 200. resp.url is the final URL; if it differs from the one requested, the detail mentions the redirect. That matters for health checks: a load balancer redirecting /health to a login page would otherwise look healthy.
  2. HTTPError. The server answered, but with 4xx or 5xx. urlopen raises instead of returning, and e.code holds the status. This clause must come before URLError, because HTTPError is a subclass of it.
  3. URLError. No HTTP response at all. Its reason tells you why: a timeout object for a connect timeout, or an OSError such as connection refused. Anything that is not a timeout is reported as unreachable.
  4. TimeoutError. If the connection succeeds but the body is slow, the timeout fires during the read and arrives as a plain TimeoutError (socket.timeout is an alias of it since Python 3.10).

The test server returns each failure mode on purpose. The /slow handler catches OSError because by the time it writes, the client has already closed the connection. The closed port is found by binding a socket to port 0, reading the port number and closing it again.

u is the number of URLs, checked one after another. The script takes about 0.5 s for the timeout plus the time the OS needs to refuse the closed port; checking many URLs in parallel is a follow-up.

ComplexityTime O(u)Space O(1)
Python
import socket
import threading
import time
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
from urllib.error import HTTPError, URLError
from urllib.request import urlopen
class Handler(BaseHTTPRequestHandler):
# Local test server with one endpoint per failure mode.
def do_GET(self):
try:
if self.path == "/healthz":
self._reply(200, b"ok")
elif self.path == "/old-health":
self.send_response(301)
self.send_header("Location", "/healthz")
self.send_header("Content-Length", "0")
self.end_headers()
elif self.path == "/broken":
self._reply(500, b"boom")
elif self.path == "/slow":
time.sleep(1.5) # longer than the client timeout
self._reply(200, b"late")
else:
self._reply(404, b"not found")
except OSError:
pass # the client gave up on /slow and closed the socket
def _reply(self, code, body):
self.send_response(code)
self.send_header("Content-Length", str(len(body)))
self.end_headers()
self.wfile.write(body)
def log_message(self, *args):
pass
def check(url, timeout=0.5):
# Returns (verdict, detail). Never raises for network problems.
try:
with urlopen(url, timeout=timeout) as resp:
detail = str(resp.status)
if resp.url != url:
detail += f" (redirected to {resp.url.split('/', 3)[-1]})"
return "UP", detail
except HTTPError as e: # server answered with 4xx/5xx
return "DOWN", str(e.code)
except URLError as e: # no HTTP answer at all
if isinstance(e.reason, (TimeoutError, socket.timeout)):
return "DOWN", "timeout"
return "DOWN", "unreachable"
except TimeoutError: # timeout while reading the body
return "DOWN", "timeout"
def closed_port():
s = socket.socket()
s.bind(("127.0.0.1", 0))
port = s.getsockname()[1]
s.close() # nothing listens here any more
return port
server = ThreadingHTTPServer(("127.0.0.1", 0), Handler)
server.daemon_threads = True
threading.Thread(target=server.serve_forever, daemon=True).start()
base = f"http://127.0.0.1:{server.server_address[1]}"
urls = [(base + p, 0.5) for p in ["/healthz", "/old-health", "/broken", "/missing", "/slow"]]
# Longer timeout here: some OSes (Windows) retry a refused connect for ~2 s before failing.
urls.append((f"http://127.0.0.1:{closed_port()}/healthz", 5))
down = 0
for url, timeout in urls:
verdict, detail = check(url, timeout)
down += verdict == "DOWN"
name = url.split("/", 3)[-1] if url.startswith(base) else "closed-port/healthz"
print(f"{verdict:<5}{detail:<30}/{name}")
print(f"{down} of {len(urls)} down")
server.shutdown()
server.server_close()

Follow-up questions

  • Check 200 URLs with at most 20 in flight at a time.
  • Treat a response slower than 300 ms as SLOW even when it succeeds.
  • Exit non-zero if any URL is down, so the script can gate a deploy.

Frequently asked questions

Deploy pipelines, smoke tests and synthetic monitors all boil down to "request this URL and decide if it is healthy". The interesting part is the failure modes: a timeout, a refused connection and a 500 mean different things when you are on call.

Without one, urlopen uses the global socket default, which is no timeout at all. A single host that accepts the connection but never answers will then hang the checker indefinitely.

It depends on the endpoint. For a dedicated /healthz path, an unexpected redirect usually means something is misconfigured. Reporting where the request ended up lets the caller decide; a stricter checker could treat any redirect as DOWN.