j4flmao/agent-skills/network-protocols

Network Protocols (Transport Layer)

Purpose

Latency spikes, stalled transfers, connection exhaustion and "slow on mobile" problems usually trace back to transport behaviour: handshakes, congestion windows, retransmission timers, Nagle/delayed-ACK interaction, MTU black holes, or TIME_WAIT. This skill explains the mechanisms with their RFCs and gives the knobs and captures to diagnose them.

Agent Protocol

Trigger

"TCP state", "TIME_WAIT", "handshake", "congestion control", "CUBIC", "BBR", "Nagle", "TCP_NODELAY", "retransmission", "MTU", "QUIC", "HTTP/3", "UDP vs TCP", "socket tuning", "tcpdump".

Input Context

• Symptom and where it appears (client, server, load balancer), RTT and loss characteristics
• OS/kernel version, protocol stack (TCP+TLS, QUIC), traffic pattern (request/response, bulk, streaming)
• Available evidence: ss -ti, netstat -s, pcap

Output Artifact

Diagnosis, socket/sysctl changes, capture/analysis commands. No file unless requested.

Response Format

Symptom: {observed}
Mechanism: {protocol behaviour responsible}
Evidence: {ss/pcap/counter lines}
Fix: {socket option / sysctl / design change}
Verify: {metric to re-measure}

1. TCP Connection Lifecycle (RFC 9293, which obsoletes RFC 793)

Client: CLOSED → SYN_SENT → ESTABLISHED → FIN_WAIT_1 → FIN_WAIT_2 → TIME_WAIT → CLOSED
Server: LISTEN → SYN_RCVD → ESTABLISHED → CLOSE_WAIT → LAST_ACK → CLOSED

• Handshake costs 1 RTT before data (plus 1 RTT for TLS 1.3, 2 for TLS 1.2). TCP Fast Open and TLS 1.3 0-RTT can trim this with replay caveats.
• TIME_WAIT is held by the side that closes first for 2×MSL (Linux: fixed 60 s) so delayed segments cannot corrupt a new connection with the same 4-tuple. Many TIME_WAIT sockets on a client exhaust ephemeral ports: reuse connections (keep-alive/pooling), widen net.ipv4.ip_local_port_range, enable net.ipv4.tcp_tw_reuse (safe for outgoing connections with timestamps). Never use the removed tcp_tw_recycle.
• CLOSE_WAIT piling up means the application never calls close() — an app bug, not a kernel setting.
• Listen backlog: SYN queue + accept queue (somaxconn, listen(fd, backlog)); overflows show in netstat -s ("SYNs to LISTEN sockets dropped").

2. Flow Control and Throughput

• Receiver advertises a window; window scaling (RFC 7323) allows > 64 KB. Throughput ≤ window / RTT, so the bandwidth-delay product (BDP = bandwidth × RTT) must fit in socket buffers: 1 Gbit/s × 100 ms ≈ 12.5 MB.
• Linux auto-tunes buffers within net.ipv4.tcp_rmem / tcp_wmem; raise the max for long fat networks rather than setting SO_RCVBUF (which disables auto-tuning).

3. Congestion Control

• Slow start doubles cwnd per RTT from an initial window of 10 segments (RFC 6928) — short transfers are RTT-bound, not bandwidth-bound.
• Switch per host: sysctl net.ipv4.tcp_congestion_control=bbr (with fq qdisc for pacing on older kernels). Measure on your paths; BBR mainly helps lossy, long-RTT links.

4. Loss Recovery

• Fast retransmit on 3 duplicate ACKs; SACK (RFC 2018) tells the sender exactly what is missing; RACK-TLP (RFC 8985) is Linux's time-based loss detection.
• RTO: minimum 200 ms on Linux (RFC 6298 suggests 1 s), doubling on each timeout — a single tail loss on a short request can add hundreds of ms; this is a classic p99 latency source.
• Inspect with ss -ti (cwnd, rtt, retrans, pacing rate) and nstat/netstat -s retransmission counters.

5. Small-Message Latency: Nagle and Delayed ACK

• Nagle holds small segments while unacknowledged data is in flight; the peer's delayed ACK (up to ~40 ms on Linux, 200 ms on some stacks) waits for more data → write-write-read patterns stall.
• Fix: TCP_NODELAY for request/response and interactive protocols (most RPC/HTTP libraries set it), or coalesce writes (writev, TCP_CORK/MSG_MORE then flush).

6. MTU and Fragmentation

• Ethernet MTU 1500 → TCP MSS 1460 (IPv4) / 1440 (IPv6) without options. Tunnels (VXLAN, WireGuard, IPsec) reduce it.
• Path MTU discovery relies on ICMP "fragmentation needed"/"packet too big"; firewalls dropping ICMP create black holes (handshake works, large responses hang). Fixes: allow ICMP, MSS clamping on tunnel endpoints, PLPMTUD (net.ipv4.tcp_mtu_probing=1).
• UDP protocols should keep datagrams ≤ ~1200 bytes unless the path is known (QUIC's minimum is 1200).

7. QUIC and HTTP/3

• QUIC (RFC 9000, TLS 1.3 integrated per RFC 9001) runs over UDP: 1-RTT handshake (0-RTT resumption), independent streams without TCP head-of-line blocking, connection migration via connection IDs, encrypted transport headers, user-space congestion control (CUBIC/BBR in the library).
• HTTP/3 (RFC 9114) maps HTTP onto QUIC; discovered via Alt-Svc or HTTPS DNS records; clients fall back to HTTP/2 over TCP when UDP is blocked.
• Costs: higher CPU per byte than kernel TCP (use GSO/GRO, UDP_SEGMENT), middleboxes that rate-limit UDP.

8. UDP vs TCP Decision

9. Diagnosis Toolkit

ss -tinp state established '( dport = :443 )'     # per-socket cwnd/rtt/retrans
nstat -az | grep -i -E 'retrans|timeout|listendrop' # counters
tcpdump -i eth0 -s 128 -w cap.pcap 'tcp port 443'   # then analyse in Wireshark (Statistics → TCP Stream Graphs)
mtr -rwzc 100 host                                  # per-hop loss/latency

In Wireshark look for retransmissions, zero windows, duplicate ACKs, long gaps after small segments (Nagle/delayed ACK) and missing large packets (MTU).

10. Anti-Patterns

• New TCP+TLS connection per request (handshake RTTs, TIME_WAIT churn) instead of pooling.
• Setting fixed SO_RCVBUF/SO_SNDBUF small values that cap throughput on high-BDP paths.
• Blocking all ICMP.
• Tuning sysctls from blog posts without measuring (e.g. tcp_tw_recycle, which broke NAT clients and was removed in Linux 4.12).
• Treating every timeout as packet loss without checking app-level stalls (CLOSE_WAIT, full accept queue).

11. Checklist

• [ ] Connections reused; idle timeouts aligned across client, LB and server
• [ ] Buffers allow the path's BDP; auto-tuning not disabled accidentally
• [ ] TCP_NODELAY or write coalescing for request/response traffic
• [ ] MTU/MSS correct through tunnels; ICMP PTB allowed
• [ ] Congestion control choice measured on real paths
• [ ] Retransmission/drop counters monitored