It is 3 AM at Swiggy. Order volume has not changed. No deployment happened in the last six hours. But checkout is failing for 8% of users, and the on-call engineer is staring at a service that "did not change." Nothing about the application changed. The network underneath it did. This is the pattern every SRE eventually learns the hard way: a surprising number of production incidents that look like application failures are actually caused by networking wearing an application costume. A connection pool quietly exhausts itself over four hours until port allocation fails. A DNS TTL that was fine at 100 requests per second causes cache stampedes at 10,000. A load balancer's least-connections algorithm sends traffic to a pod that is technically alive but backed up on a slow database query. You already know what a subnet is and how `ping` works. This module assumes that. What it teaches is the layer above - the behavior of TCP, DNS, TLS, and load balancers specifically **under production load**, and the exact commands you run at 3 AM to find out which one is lying to you. > 📌 **Remember:** In distributed systems, "it's probably the network" is usually correct. The skill is proving *which part* of the network, fast. -
A TCP connection is not just "open" or "closed." It moves through a defined sequence of states, and production incidents live in the transitions between them. If you only know `ESTABLISHED` and `CLOSED`, you are blind to the two states that cause the most real outages: `TIME_WAIT` and `CLOSE_WAIT`. ### The full connection lifecycle CLIENT SERVER SYN_SENT ----SYN---------> LISTEN <---SYN-ACK------ SYN_RECEIVED ESTABLISHED ----ACK---------> ESTABLISHED <====data flows both ways====> -- closing (client-initiated) -- FIN_WAIT_1 ----FIN---------> CLOSE_WAIT FIN_WAIT_2 <---ACK---------- <---FIN---------- LAST_ACK TIME_WAIT ----ACK---------> CLOSED (60s wait) CLOSED The handshake to open a connection is three steps (SYN, SYN-ACK, ACK). Closing takes four, because each side must independently confirm it has finished sending. That asymmetry is where the two dangerous states come from. ### TIME_WAIT - normal, but dangerous at volume `TIME_WAIT` appears on whichever side sent the first `FIN` - the active closer. The kernel holds the connection's port for roughly 2 x MSL (Maximum Segment Lifetime, about 60 seconds on Linux) so that any stray, delayed packet from the old connection cannot be misread as part of a *new* connection reusing the same port. <cite index="5-1">It is not inefficiency - it is TCP's final act of reliability.</cite> The problem is not that `TIME_WAIT` exists. The problem is what happens when your service opens thousands of short-lived outbound connections per second - for example, a service that opens a fresh HTTP connection per request instead of reusing a connection pool. ```bash ## Count connections currently in TIME_WAIT ss -Htn state time-wait | wc -l ## Watch it climb - if it keeps growing, you have a leak pattern, not just volume watch -n 5 "ss -Htn state time-wait | wc -l" ``` > **Note:** Each local `(source IP, source port)` pair used for an outbound connection cannot be reused until its `TIME_WAIT` timer expires. With only ~28,000 usable ephemeral ports by default, a service opening 500 short connections/sec to the same downstream host can exhaust its entire port range in under a minute - new outbound connections then fail with `EADDRNOTAVAIL`, even though the destination is perfectly healthy. ```bash ## Check your ephemeral port range sysctl net.ipv4.ip_local_port_range ## Example output: 32768 60999 -> about 28,000 ports available ## Safe production mitigation: widen the port range sudo sysctl -w net.ipv4.ip_local_port_range="10000 65000" ## tcp_tw_reuse allows reusing a TIME_WAIT socket for a NEW outbound connection ## when it is safe to do so (timestamps enabled). This is the standard fix - ## NOT tcp_tw_recycle, which is removed in modern kernels and was unsafe behind NAT. sudo sysctl -w net.ipv4.tcp_tw_reuse=1 ``` > 🔴 **Common Mistake:** Reaching for `tcp_tw_recycle` because an old blog post recommends it. That flag was removed from the Linux kernel because it silently broke connections from clients behind NAT (multiple users sharing one public IP get randomly rejected). Use `tcp_tw_reuse` and, more importantly, fix the real problem - use connection pooling so you are not opening thousands of short connections in the first place. > 📌 **Remember:** Kernel tuning is a mitigation, not the fix. `tcp_tw_reuse` and a wider port range buy you headroom, but they do not stop a service from opening one connection per request. The actual fix - the one that removes the problem instead of tolerating it - is connection pooling at the application layer. If your only takeaway from this section is "enable tcp_tw_reuse," you have learned the wrong lesson. ### CLOSE_WAIT - never normal at scale `CLOSE_WAIT` is the mirror image. It appears on whichever side *received* the `FIN` first - meaning the remote end has already hung up, the kernel has acknowledged it, and now it is waiting for **your application code** to call `close()` on the socket. <cite index="7-1">CLOSE_WAIT means the remote side has closed the connection, but the local application has not yet called close() on the socket - this is almost always an application bug, the code is holding onto connections it should have released.</cite> <cite index="8-1">Persistent CLOSE_WAIT almost always indicates an application bug where code is not closing connections after the remote end has finished.</cite> NORMAL (transient): STUCK (leak): Remote sends FIN Remote sends FIN -> local ACKs -> CLOSE_WAIT -> local ACKs -> CLOSE_WAIT -> app calls close() -> app never calls close() -> LAST_ACK -> CLOSED -> stays CLOSE_WAIT forever (milliseconds) (grows without bound) ```bash ## Count CLOSE_WAIT connections ss -Htn state close-wait | wc -l ## Find which PROCESS is holding them open ss -tnp state close-wait ## Extract the PID and inspect its open file descriptors ss -tnp state close-wait | grep -oP 'pid=\K\d+' ls -la /proc/<pid>/fd | grep socket ``` > 🔴 **Common Mistake:** Confusing `TIME_WAIT` and `CLOSE_WAIT` because both have "WAIT" in the name. <cite index="10-1">TIME_WAIT exists on the side that actively closes by sending FIN first, while CLOSE_WAIT exists on the side that receives the FIN from the remote end</cite> - they represent opposite roles and opposite fixes. A high `TIME_WAIT` count is a *volume* problem you tune around. A growing `CLOSE_WAIT` count is a *bug* you patch: use context managers or `try/finally` blocks so `close()` always runs, even on exceptions. ### Quick reference - reading `ss -tan` output | Symptom | Meaning | Where to look | |:---|:---|:---| | High and stable `TIME_WAIT` | Normal churn from many short connections | Connection pooling config | | `TIME_WAIT` climbing with `EADDRNOTAVAIL` errors | Ephemeral port exhaustion | `tcp_tw_reuse`, port range, pooling | | `CLOSE_WAIT` growing over hours | Application not calling `close()` | App code - exception handling around sockets | | `SYN_RECV` spiking | Handshake flood or backlog overflow | `net.core.somaxconn`, possible SYN flood | | `ESTABLISHED` count flat but errors rising | Connections open but app-level failure | Check app logs, not TCP layer | -
`TIME_WAIT` and `CLOSE_WAIT` are what you see after a connection has been through its lifecycle. This section is about the moment a connection is *born* - the handshake queues that decide whether a new connection gets accepted at all, and the settings that decide how efficiently data moves once it is open. ### Two queues, not one Every listening socket actually has two separate queues, and confusing them wastes debugging time. Client SYN arrives | v +-------------------+ size limited by: | SYN queue | min(somaxconn-derived limit, tcp_max_syn_backlog) | (half-open conns, | | waiting for ACK) | +-------------------+ | final ACK received v +-------------------+ size limited by: | Accept queue | min(listen() backlog argument, net.core.somaxconn) | (fully established,| | waiting for the | | app to accept()) | +-------------------+ | app calls accept() v Handed to application <cite index="35-1">The SYN queue holds connections where a SYN has been received and a SYN-ACK sent, but no ACK has come back yet - these are lightweight "request sockets," not full connections. The accept queue holds fully established connections waiting for the application to call accept()</cite>, and <cite index="39-1">when either queue fills up, incoming SYNs are silently dropped, causing connection timeouts from the client's perspective</cite> - not a clean rejection, a hang. ### Why this matters at 3 AM A client-side symptom of a full accept queue looks identical to "the server is slow" - the client just times out waiting for a response that never comes, because the kernel never even handed the connection to the application. This is a classic case of blaming application code for what is actually a kernel queue limit. ```bash ## Check current queue depth vs. capacity for every listening socket ss -tlnp ## Recv-Q = current accept queue size (connections waiting for accept()) ## Send-Q = the configured backlog limit for that socket ## Recv-Q approaching Send-Q means the application is not accepting fast enough ## Confirm whether SYNs or established connections are actually being dropped netstat -s | grep -i "listen\|overflowed" ## "SYNs to LISTEN sockets dropped" -> SYN queue overflow ## "times the listen queue of a socket overflowed" -> accept queue overflow ``` <cite index="39-1">The SYN queue defaults to 128-1024 depending on the system, while the accept queue defaults to the smaller of the application's listen backlog and net.core.somaxconn.</cite> <cite index="36-1">The somaxconn default (128 on older kernels, 4096 on Linux 5.4+) is often too low for production servers under heavy load.</cite> ```bash ## Raise both queue limits - the application's listen() backlog argument ## must also be raised, or it becomes the new bottleneck regardless of this setting sudo sysctl -w net.core.somaxconn=65535 sudo sysctl -w net.ipv4.tcp_max_syn_backlog=65535 ``` > 🔴 **Common Mistake:** Raising `net.core.somaxconn` and assuming the problem is solved. If the application's own `listen(fd, backlog)` call still passes a low number (many frameworks default to 128), the kernel still uses `min(backlog, somaxconn)` - the kernel setting did nothing because the application-level value is the smaller of the two. Check both. ### Socket buffers and keepalive ```bash ## Socket buffer sizes control how much data can be in flight before the ## sender blocks waiting for the receiver to catch up - undersized buffers ## throttle throughput on high-latency links even when bandwidth is available sysctl net.core.rmem_max net.core.wmem_max sysctl net.ipv4.tcp_rmem net.ipv4.tcp_wmem ## TCP keepalive detects a dead peer when no FIN/RST ever arrives - for ## example, the remote host lost power or a middlebox silently dropped state sysctl net.ipv4.tcp_keepalive_time ## seconds of idle before probing starts (default 7200 - often too long for production) sysctl net.ipv4.tcp_keepalive_intvl ## seconds between probes sysctl net.ipv4.tcp_keepalive_probes ## probes sent before declaring the connection dead ``` > **Note:** The default `tcp_keepalive_time` of 7200 seconds (2 hours) means a genuinely dead connection can sit in `ESTABLISHED` state, silently consuming a slot, for two hours before the kernel notices. For a service holding long-lived connections (database pools, gRPC streams), lowering this to something like 60-300 seconds means dead peers get cleaned up in minutes instead of hours - directly preventing the kind of silent connection accumulation that eventually starves a pool. ### Quick reference - TCP tuning parameters | Parameter | Controls | Production default worth considering | |:---|:---|:---| | `net.core.somaxconn` | Max accept queue size | 65535 (must match app's `listen()` backlog) | | `net.ipv4.tcp_max_syn_backlog` | Max SYN (half-open) queue size | 65535 under heavy inbound load | | `net.ipv4.tcp_keepalive_time` | Idle time before keepalive probes start | 60-300s for long-lived connection services | | `net.core.rmem_max` / `wmem_max` | Max socket buffer size | Raise for high-bandwidth, high-latency links | | `net.ipv4.tcp_tw_reuse` | Allow reusing TIME_WAIT sockets for new outbound connections | 1, alongside connection pooling | -
DNS rarely fails as "it does not work." It fails as "it works for some users, some of the time, and only during traffic spikes." That intermittency is what makes DNS incidents expensive - by the time someone notices, the cache that would explain it has already expired. ### The resolution path, and where it silently breaks App calls resolve("payments-db.internal") | v Local resolver cache (nscd / systemd-resolved) --- expired? continue | v /etc/resolv.conf -> configured nameserver (e.g. CoreDNS in-cluster) | v Nameserver cache --- MISS -> recursive query outward | v Authoritative answer returned, cached per its TTL | v Answer returned to app Every hop above has its own cache, its own TTL, and its own failure mode. When someone says "DNS is being weird," your first job is to figure out *which hop*. ### TTL and the cache stampede A short TTL (say, 30 seconds) is common for services that need fast failover - if an IP changes, clients should notice quickly. But a short TTL on a **high-traffic** name means that name gets re-resolved constantly. If the authoritative server or resolver has a brief hiccup at the exact moment thousands of clients try to refresh simultaneously, you get a **cache stampede**: a wave of clients with expired entries hitting the resolver at once, all timing out together, producing a burst of failures that looks exactly like an application outage. ```bash ## Check the TTL currently being served for a name dig +noall +answer payments-db.internal ## payments-db.internal. 30 IN A 10.0.2.14 ## ^-- TTL in seconds ## Query a specific resolver directly, bypassing local cache - ## use this to tell "my cache is stale" apart from "the real answer changed" dig @10.0.0.10 payments-db.internal ## Negative caching - NXDOMAIN gets cached too, per the SOA record's minimum TTL ## This is why a typo'd hostname can "stay broken" for minutes after you fix DNS dig +noall +authority nonexistent.internal ``` > 💡 **Tip:** Before any planned IP change - a database failover, a load balancer swap - lower the TTL on that record hours ahead of time. A TTL of 300 (5 minutes) that has been live for 24+ hours means the cutover propagates in minutes. A TTL you drop 5 minutes before the change does nothing, because clients are still holding the old, longer-TTL answer in cache. ### DNS inside Kubernetes - CoreDNS specifics Kubernetes runs its own DNS layer (CoreDNS) for service discovery, and it has failure modes the wider internet does not. ```bash ## Confirm DNS resolution from INSIDE a pod - the most common first step kubectl exec -it my-pod -- nslookup payments-service.default.svc.cluster.local ## Check CoreDNS itself is healthy kubectl get pods -n kube-system -l k8s-app=kube-dns kubectl logs -n kube-system -l k8s-app=kube-dns --tail=100 ## A very common silent failure: CoreDNS CPU throttled under load, ## causing slow (not failed) lookups that look like app latency kubectl top pods -n kube-system -l k8s-app=kube-dns ``` > 🔴 **Common Mistake:** Chasing application-level timeout tuning when the real problem is CoreDNS being CPU-throttled during a traffic spike. A DNS lookup that takes 4 seconds instead of 4 milliseconds looks exactly like a slow database from the application's point of view - always rule out DNS latency before assuming the downstream service is slow. -
HTTPS failing is one of the most common "everything is on fire" pages, and one of the fastest to diagnose once you know which step of the handshake broke. ### What actually happens during a TLS handshake CLIENT SERVER ClientHello (supported ciphers, SNI) ---> <--- ServerHello (chosen cipher) <--- Certificate (chain + public key) <--- ServerHelloDone [verify certificate chain locally] [verify hostname matches cert / SNI] ClientKeyExchange --------------------> [both sides derive shared session key] Finished (encrypted) -----------------> <--- Finished (encrypted) == encrypted application data flows == Every TLS error you will ever see maps to one specific step above failing. Learning to read the error message tells you exactly where to look. ### Diagnosing from the outside ```bash ## Full handshake dump - shows the certificate chain, protocol version, cipher openssl s_client -connect api.payments.internal:443 -showcerts ## The single most important production check: certificate expiry echo | openssl s_client -connect api.payments.internal:443 2>/dev/null \ | openssl x509 -noout -dates ## notAfter=Nov 12 23:59:59 2026 GMT ## An expired cert does not degrade gracefully - every client refuses the ## connection outright, so this looks like a total outage, not a warning ``` > ⚠️ **Security:** Certificate expiry is one of the few outages you can predict weeks in advance and still miss. Alert on certificates expiring within 30 days, not the day they actually expire - renewal, testing, and rollout all take time you will not have during an active incident. ### Reading the actual error against the handshake step | Error message | Handshake step that failed | What to check | |:---|:---|:---| | `certificate has expired` | Client verifying certificate | Renew and redeploy the cert | | `unable to get local issuer certificate` | Client verifying the chain | Missing intermediate certificate on server | | `certificate name mismatch` / SNI error | Hostname verification | Cert does not cover the SNI hostname requested | | `connection reset during handshake` | ServerHello / ClientKeyExchange | Load balancer terminating TLS incorrectly, or cipher mismatch | | `handshake timeout` | ClientHello never answered | Firewall or security group blocking port 443, not a TLS problem at all | > 🔴 **Common Mistake:** Treating every HTTPS failure as a "TLS problem" and reaching for `openssl s_client` first. A handshake timeout with no response at all is almost always a network path problem (security group, network policy) - TLS never even started. Confirm you can reach the port first with a plain TCP check before debugging certificates. -
Every generation of HTTP solved the previous generation's biggest bottleneck, and every fix moved the bottleneck somewhere new rather than eliminating it. Knowing which generation your traffic uses changes how you interpret a "slow request" report. ### HTTP/1.1 - one request, one turn, per connection <cite index="28-1">HTTP/1.1 had head-of-line blocking because it needs to send its responses in full and cannot multiplex them</cite> over a single connection. <cite index="30-1">In HTTP/1.1, browsers opened up to six connections per domain, but within each connection, requests queued serially - request #2 could not start until request #1 completed, even if #2's resource was ready first.</cite> Keep-alive reduced handshake overhead by reusing one connection for multiple sequential requests, but sequential is the key word - it did not fix the queuing. HTTP/1.1 on ONE connection: Request A ----[waiting]----> Response A Request B ----[waiting]----> Response B (B cannot even start until A fully finishes) ### HTTP/2 - multiplexing fixes the application layer, not the transport layer <cite index="27-1">HTTP/2 introduced streams - multiple logical channels that allow concurrent requests and responses over a single TCP connection</cite>, which removed the application-layer queuing HTTP/1.1 suffered from. But <cite index="30-1">HTTP/2 introduced a deeper problem at the transport layer: TCP guarantees in-order delivery of bytes, so when one packet drops on a connection carrying 100 multiplexed streams, TCP's receive buffer holds the later packets and refuses to deliver them to the application until the lost packet is retransmitted - stalling all 100 streams even though 99 of them have no dependency on the lost packet.</cite> HTTP/2 - multiplexed over ONE TCP connection: Stream 1, Stream 2, Stream 3 all interleaved on the wire | v [ packet loss on ANY stream ] | v TCP holds EVERYTHING until retransmit completes (all streams stall, even the ones with no lost data) > 🔴 **Common Mistake:** Assuming HTTP/2 is always faster than HTTP/1.1 because "multiplexing." <cite index="28-1">On slower networks with higher packet loss, HTTP/1.1 running six separate connections can outperform HTTP/2 running on one connection</cite>, precisely because a single lost packet on the one HTTP/2 connection stalls every request sharing it - a client on a lossy mobile network is a real scenario where this shows up as unexplained latency spikes. ### HTTP/3 - moving multiplexing into the transport itself <cite index="25-1">HTTP/3 abandons TCP entirely and adopts QUIC, built on UDP, where each QUIC stream handles packet loss independently - an image request's packet loss does not affect an unrelated JS file's loading.</cite> QUIC implements the concept of independent streams natively at the transport layer, so a lost packet on one stream no longer blocks unrelated streams sharing the same connection. HTTP/3 over QUIC (UDP): Stream 1 [packet lost, retransmitting] -- stalls only Stream 1 Stream 2 [flowing normally] -- unaffected Stream 3 [flowing normally] -- unaffected > **Note:** QUIC's head-of-line blocking fix only helps when multiple streams are actually active at the same moment. A single active stream with packet loss stalls exactly like TCP would, because there is nothing else to make progress on while it waits. ### Why this matters for diagnosis ```bash ## Confirm which protocol version a connection actually negotiated - ## this changes what "one slow request is blocking others" even means curl -v --http2 https://api.payments.internal/health 2>&1 | grep -i "http/" curl -v --http3 https://api.payments.internal/health 2>&1 | grep -i "http/" ``` | Protocol | Multiplexing location | HOL blocking source | Debugging implication | |:---|:---|:---|:---| | HTTP/1.1 | None (sequential, or 6 parallel connections) | Application layer, per connection | A slow request blocks only its own connection's queue | | HTTP/2 | Application layer (streams), one TCP connection | Transport layer (TCP in-order delivery) | One dropped packet can stall every request on that connection | | HTTP/3 | Native, over QUIC (UDP) | Eliminated per-stream, except when only one stream is active | Packet loss on one request no longer stalls unrelated requests | > 💡 **Tip:** If you see "one endpoint is slow and everything behind it on the same connection gets slow too" on HTTP/2 traffic, suspect transport-layer packet loss (check with `tcpdump` for retransmissions) before suspecting the endpoint itself. This exact symptom does not happen the same way on HTTP/3. -
It is 3 AM at Swiggy. Order volume has not changed. No deployment happened in the last six hours. But checkout is failin...
A TCP connection is not just "open" or "closed." It moves through a defined sequence of states, and production incidents...
TIMEWAIT and CLOSEWAIT are what you see after a connection has been through its lifecycle. This section is about the mom...
DNS rarely fails as "it does not work." It fails as "it works for some users, some of the time, and only during traffic ...
HTTPS failing is one of the most common "everything is on fire" pages, and one of the fastest to diagnose once you know ...
Every generation of HTTP solved the previous generation's biggest bottleneck, and every fix moved the bottleneck somewhe...
Everything covered so far applies to a single host talking to another single host. Inside a Kubernetes cluster, every on...
Kubernetes ships with three built-in probe types - HTTP, TCP, and exec - and every one of them is the wrong tool for che...
Every load balancing algorithm makes a trade-off, and every trade-off has a scenario where it actively makes things wors...
These four tools cover the overwhelming majority of production network diagnosis. Each answers a different layer of the ...
Work through these in order. Each step produces evidence you would actually pull during a real incident. Reproduce TIMEW...
Symptom Likely cause First command to run Outbound requests fail with EADDRNOTAVAIL Ephemeral port exhaustion from TIMEW...
Engineers reach for tcptwrecycle when they see high TIMEWAIT counts because older tutorials still recommend it. That fla...
Aligns directly with DevOps, Site Reliability (SRE), and Platform Engineering job descriptions.