You already know: processes talk to each other through the OS, and waiting on I/O doesn't need the CPU (Operating systems §1, §2, §6).
Why this matters for Atlas. Atlas has to be reachable from any browser. Every answer crosses the internet, so the network adds delay, can fail halfway, and can be overwhelmed. Good networking turns "works on my laptop" into "works for 10,000 students".
1. The journey of one request
The question. What happens between pressing Enter and seeing the first word of an answer?
Start from nothing. Data travels in layers, each solving one problem:
| Step | Protocol | Problem it solves |
|---|---|---|
| 1 | DNS | Turn api.atlas.dev into an IP address (a machine's number on the internet) |
| 2 | IP | Route packets hop by hop across networks to that address |
| 3 | TCP | Make delivery reliable and ordered: number packets, resend lost ones |
| 4 | TLS | Encrypt and authenticate, so nobody in between can read or fake the data |
| 5 | HTTP | Say what you want: POST /ask with a JSON body; get a status code and a response |
Connecting costs round trips (RTTs) before any data flows: roughly one for the TCP handshake and one more for a TLS 1.3 handshake. That's why reusing connections matters (section 4).
Real-life analogy. Posting a tracked, sealed letter: look up the address (DNS), the postal network routes it (IP), tracking guarantees arrival (TCP), the sealed envelope keeps it private (TLS), and the letter's text is the actual request (HTTP).
Worked example — a real TCP conversation on your own machine:
import socket
import threading
def server(sock):
conn, _ = sock.accept()
with conn:
question = conn.recv(1024).decode()
conn.sendall(f"Atlas heard: {question}".encode())
listener = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
listener.bind(("127.0.0.1", 0)) # port 0 = let the OS choose a free port
listener.listen()
port = listener.getsockname()[1]
threading.Thread(target=server, args=(listener,), daemon=True).start()
with socket.create_connection(("127.0.0.1", port)) as client: # TCP handshake happens here
client.sendall(b"What is a page table?")
print(client.recv(1024).decode())
listener.close()
Common trap. "HTTPS is slow." The encryption itself is cheap on modern CPUs; the cost is the extra handshake round trip — which connection reuse removes.
2. Latency vs bandwidth
The question. Why does a tiny request to a far-away server feel slow even on fast internet?
Start from nothing. Two different numbers describe a network:
- Latency — how long one bit takes to arrive (set by distance and hops). Light in fibre covers about 200 km per millisecond, so physics alone puts a floor under round trips between continents.
- Bandwidth — how many bits per second fit through the pipe.
Time to receive a response ≈ round trips × RTT + size ÷ bandwidth. Small requests are dominated by latency; big downloads by bandwidth.
Real-life analogy. Latency is how long the road is; bandwidth is how many lanes it has. More lanes don't make a single car arrive sooner.
Worked example.
def transfer_ms(size_bytes, bandwidth_mbps, rtt_ms, round_trips):
return round_trips * rtt_ms + size_bytes * 8 / (bandwidth_mbps * 1e6) * 1000
cases = {
"1 KB question, same city (RTT 5 ms), new connection": (1_000, 100, 5, 3),
"1 KB question, other continent (RTT 150 ms), new connection": (1_000, 100, 150, 3),
"1 KB question, other continent, reused connection": (1_000, 100, 150, 1),
"2 GB model download, same city": (2_000_000_000, 100, 5, 3),
}
for name, args in cases.items():
print(f"{transfer_ms(*args):>10.1f} ms {name}")
In production / interviews. For chat apps the key metric is time to first token; streaming (section 3) and nearby regions/edges cut it. Interview question: "Our users in another continent see 600 ms before the first token. Where would you look?"
3. Choosing an API style: REST, gRPC, streaming
The question. How should a client and an AI service talk — and how do we send an answer word by word?
Start from nothing.
- REST over HTTP + JSON — resources and verbs (
POST /v1/answers). Human-readable, works everywhere. The default for public APIs. - gRPC — functions defined in a schema (Protocol Buffers), compact binary messages, running over HTTP/2 (many requests multiplexed over one connection). Common between internal services.
- Streaming the answer. LLMs produce tokens one at a time. Instead of waiting for the full reply, servers stream:
- Server-Sent Events (SSE) — one HTTP response that stays open and sends
data:lines. Simple; used by most LLM APIs. - WebSockets — a two-way channel, for when the client also sends continuously (voice, live collaboration).
- Server-Sent Events (SSE) — one HTTP response that stays open and sends
- HTTP/3 runs HTTP over QUIC (on UDP), avoiding TCP's head-of-line blocking and making connection setup faster.
Worked example — what an SSE token stream looks like on the wire:
import json
import time
def sse_stream(tokens):
for tok in tokens:
time.sleep(0.05) # the model generating this token
yield f"data: {json.dumps({'token': tok})}\n\n"
yield "data: [DONE]\n\n"
start = time.perf_counter()
answer = []
for i, event in enumerate(sse_stream(["A", " page", " table", " maps", " pages", "."])):
payload = event.removeprefix("data: ").strip()
if payload == "[DONE]":
break
if i == 0:
print(f"first token after {(time.perf_counter() - start) * 1000:.0f} ms") # ~50 ms
answer.append(json.loads(payload)["token"])
print(f"full answer after {(time.perf_counter() - start) * 1000:.0f} ms: {''.join(answer)}") # ~300 ms
# Streaming shows text after ~50 ms; waiting for the whole reply would show nothing for ~300 ms.
A minimal Atlas endpoint (FastAPI) — shown for shape; needs pip install fastapi uvicorn:
from fastapi import FastAPI
from fastapi.responses import StreamingResponse
from pydantic import BaseModel
app = FastAPI()
class Question(BaseModel):
text: str
@app.post("/v1/ask")
async def ask(q: Question):
async def stream():
for token in ["Paging", " splits", " memory", " into", " pages", "."]:
yield f"data: {token}\n\n"
return StreamingResponse(stream(), media_type="text/event-stream")
@app.get("/healthz")
def health():
return {"status": "ok"}
| Choose | When |
|---|---|
| REST + JSON | Public APIs, browsers, simplicity |
| gRPC | Internal service-to-service, strict schemas, high call volume |
| SSE | Server → client token streaming |
| WebSocket | Two-way real-time traffic |
4. Load balancing, reverse proxies and connection pools
The question. One server can handle 200 requests at a time. Atlas gets 2,000. Now what?
Start from nothing. Run many servers and put a load balancer in front to spread requests (round-robin, least-connections, or by load). A reverse proxy (nginx, Envoy) sits in front of servers too and handles TLS, compression, routing by URL and health checks, so app servers only see plain, healthy traffic.
On the client side, opening a new TCP + TLS connection per request costs round trips. A connection pool keeps a few connections open and reuses them.
Real-life analogy. A restaurant host seating guests at whichever waiter has the fewest tables, and skipping the waiter who's on break.
Worked example — least-connections balancing:
import random
random.seed(1)
servers = {"gpu-a": 0, "gpu-b": 0, "gpu-c": 0} # active requests per server
healthy = {"gpu-a", "gpu-b", "gpu-c"}
healthy.discard("gpu-c") # failed its health check
def pick():
return min((s for s in servers if s in healthy), key=lambda s: servers[s])
for request in range(10):
s = pick()
servers[s] += 1
if random.random() < 0.5: # some requests finish quickly
servers[s] -= 1
print(servers) # gpu-c never receives traffic
In production. LLM servers are uneven — one long generation can hog a GPU — so smarter routing uses queue length or KV-cache usage, not just connection counts (D12).
5. Timeouts, retries and rate limiting
The question. A model call hangs for 60 seconds, or fails 1 time in 50. How do we stay responsive without making things worse?
Start from nothing.
- Timeout — never wait forever; give up after a budget and report an error (or fall back).
- Retry with exponential backoff and jitter — retry transient failures after 0.1 s, 0.2 s, 0.4 s… plus randomness, so thousands of clients don't retry in sync and flatten the server (thundering herd). Only retry operations that are safe to repeat (idempotent), or send an idempotency key.
- Rate limiting — cap how fast each user can call you. The classic token bucket: a bucket holds up to B tokens and refills at R per second; each request spends one token; an empty bucket means
429 Too Many Requests.
Real-life analogy. Calling a busy shop: wait a bit longer before each redial (backoff), and don't all redial at the same second (jitter). A club bouncer letting in a few people per minute (rate limit).
Worked example.
import random
import time
random.seed(7)
def flaky_model_call(attempt):
if attempt < 2:
raise TimeoutError("model busy")
return "answer"
def call_with_retries(max_attempts=5, base=0.05, cap=1.0):
for attempt in range(max_attempts):
try:
return flaky_model_call(attempt)
except TimeoutError as err:
delay = random.uniform(0, min(cap, base * 2 ** attempt)) # "full jitter"
print(f"attempt {attempt + 1} failed ({err}); retrying in {delay:.3f}s")
time.sleep(delay)
raise RuntimeError("gave up")
print(call_with_retries())
class TokenBucket:
def __init__(self, rate_per_s, burst):
self.rate, self.capacity = rate_per_s, burst
self.tokens, self.last = float(burst), time.monotonic()
def allow(self):
now = time.monotonic()
self.tokens = min(self.capacity, self.tokens + (now - self.last) * self.rate)
self.last = now
if self.tokens >= 1:
self.tokens -= 1
return True
return False
bucket = TokenBucket(rate_per_s=5, burst=3)
print(["ok" if bucket.allow() else "429" for _ in range(6)]) # 3 ok, then 429s
time.sleep(0.4)
print("after 0.4s:", "ok" if bucket.allow() else "429") # refilled ~2 tokens
Common traps
- Retrying non-idempotent calls (charging a card twice).
- Timeouts longer than the caller's timeout — work continues after nobody is waiting.
- Retrying on
429without waiting for the server'sRetry-After.
In production / interviews. Interview question: "Design rate limiting for an LLM API priced per token, where one request can be 10 tokens or 100,000." (Hint: limit tokens per minute, not just requests.)
From production
A model API sometimes takes 30 seconds under load, and clients retry immediately on timeout, all at once.
What client-side change prevents a retry storm?
Show the answer
C. Retries with exponential backoff and jitter, a retry budget, and idempotency keys so repeated requests don't duplicate work — Synchronized immediate retries multiply load on an already struggling server; backoff with jitter spreads them out and budgets cap them.
Check yourself
- Recall: Which layer guarantees packets arrive in order — IP, TCP or HTTP?
- Apply: Should a chat UI wait for the full answer or stream it? Which mechanism?
- Predict: A bucket with burst 3 and rate 5/s receives 6 instant requests. How many succeed?
Answers
- TCP.
- Stream it, usually with Server-Sent Events — it cuts time to first token.
- 3 (the burst); the rest get 429 until tokens refill.
Exercise
Write resilient_call(fn, timeout_s, max_attempts) that runs fn in a thread, gives up on any attempt longer than timeout_s, retries with full-jitter backoff, and never exceeds a total budget of timeout_s × max_attempts. Test it with a function that sleeps randomly between 0 and 0.3 s.
Solution
import random
import time
from concurrent.futures import ThreadPoolExecutor, TimeoutError as FutureTimeout
def resilient_call(fn, timeout_s=0.15, max_attempts=4, base=0.02):
deadline = time.monotonic() + timeout_s * max_attempts
with ThreadPoolExecutor(max_workers=max_attempts) as pool:
for attempt in range(max_attempts):
remaining = deadline - time.monotonic()
if remaining <= 0:
break
future = pool.submit(fn)
try:
return future.result(timeout=min(timeout_s, remaining))
except FutureTimeout:
time.sleep(random.uniform(0, base * 2 ** attempt))
raise TimeoutError("all attempts timed out")
random.seed(3)
def slow_model():
time.sleep(random.uniform(0, 0.3))
return "answer"
try:
print(resilient_call(slow_model))
except TimeoutError as e:
print(e)
This unlocks:
- Databases — where Atlas stores what the request needs.
- D11 Model deployment and D12 Inference fundamentals — streaming, routing, time to first token.
- D14 Distributed systems — retries, idempotency and failure as normal events.
Sources: Kurose & Ross, Computer Networking: A Top-Down Approach; RFC 8446 (TLS 1.3), RFC 9114 (HTTP/3); WHATWG HTML spec — Server-sent events; AWS Architecture Blog, "Exponential Backoff and Jitter"; gRPC docs.