VFlash: Fast, Accurate Visual Planning via Flash Primitive Decoding
u_Unrolled
D1: CS & Systems Foundations · Networking — How a Question Travels to the Model and Back
Phase 1/Domain 1/Lesson 20

Networking — How a Question Travels to the Model and Back

10 min read·Foundational·Free lesson
Networking — How a Question Travels to the Model and Back

You already know: processes talk to each other through the OS, and waiting on I/O doesn't need the CPU (Operating systems §1, §2, §6).

Why this matters for Atlas. Atlas has to be reachable from any browser. Every answer crosses the internet, so the network adds delay, can fail halfway, and can be overwhelmed. Good networking turns "works on my laptop" into "works for 10,000 students".


1. The journey of one request

The question. What happens between pressing Enter and seeing the first word of an answer?

Start from nothing. Data travels in layers, each solving one problem:

Step Protocol Problem it solves
1 DNS Turn api.atlas.dev into an IP address (a machine's number on the internet)
2 IP Route packets hop by hop across networks to that address
3 TCP Make delivery reliable and ordered: number packets, resend lost ones
4 TLS Encrypt and authenticate, so nobody in between can read or fake the data
5 HTTP Say what you want: POST /ask with a JSON body; get a status code and a response

Connecting costs round trips (RTTs) before any data flows: roughly one for the TCP handshake and one more for a TLS 1.3 handshake. That's why reusing connections matters (section 4).

Sequence diagram: Client ↔ DNS (lookup), Client ↔ Server (TCP SYN, SYN-ACK, ACK), (TLS hello exchange), (HTTP request → streamed response). Each RTT drawn as a diagonal arrow pair, labelled with time.
Diagram— Sequence diagram: Client ↔ DNS (lookup), Client ↔ Server (TCP SYN, SYN-ACK, ACK), (TLS hello exchange), (HTTP request → streamed response). Each RTT drawn as a diagonal arrow pair, labelled with time.

Real-life analogy. Posting a tracked, sealed letter: look up the address (DNS), the postal network routes it (IP), tracking guarantees arrival (TCP), the sealed envelope keeps it private (TLS), and the letter's text is the actual request (HTTP).

Worked example — a real TCP conversation on your own machine:

python
import socket
import threading

def server(sock):
    conn, _ = sock.accept()
    with conn:
        question = conn.recv(1024).decode()
        conn.sendall(f"Atlas heard: {question}".encode())

listener = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
listener.bind(("127.0.0.1", 0))          # port 0 = let the OS choose a free port
listener.listen()
port = listener.getsockname()[1]
threading.Thread(target=server, args=(listener,), daemon=True).start()

with socket.create_connection(("127.0.0.1", port)) as client:   # TCP handshake happens here
    client.sendall(b"What is a page table?")
    print(client.recv(1024).decode())
listener.close()

Common trap. "HTTPS is slow." The encryption itself is cheap on modern CPUs; the cost is the extra handshake round trip — which connection reuse removes.


2. Latency vs bandwidth

The question. Why does a tiny request to a far-away server feel slow even on fast internet?

Start from nothing. Two different numbers describe a network:

  • Latency — how long one bit takes to arrive (set by distance and hops). Light in fibre covers about 200 km per millisecond, so physics alone puts a floor under round trips between continents.
  • Bandwidth — how many bits per second fit through the pipe.

Time to receive a response ≈ round trips × RTT + size ÷ bandwidth. Small requests are dominated by latency; big downloads by bandwidth.

a pipe drawn twice — long and thin (high latency, low bandwidth) vs short and wide (low latency, high bandwidth) — with water droplets for bits.
Diagram— a pipe drawn twice — long and thin (high latency, low bandwidth) vs short and wide (low latency, high bandwidth) — with water droplets for bits.

Real-life analogy. Latency is how long the road is; bandwidth is how many lanes it has. More lanes don't make a single car arrive sooner.

Worked example.

python
def transfer_ms(size_bytes, bandwidth_mbps, rtt_ms, round_trips):
    return round_trips * rtt_ms + size_bytes * 8 / (bandwidth_mbps * 1e6) * 1000

cases = {
    "1 KB question, same city (RTT 5 ms), new connection": (1_000, 100, 5, 3),
    "1 KB question, other continent (RTT 150 ms), new connection": (1_000, 100, 150, 3),
    "1 KB question, other continent, reused connection": (1_000, 100, 150, 1),
    "2 GB model download, same city": (2_000_000_000, 100, 5, 3),
}
for name, args in cases.items():
    print(f"{transfer_ms(*args):>10.1f} ms  {name}")

In production / interviews. For chat apps the key metric is time to first token; streaming (section 3) and nearby regions/edges cut it. Interview question: "Our users in another continent see 600 ms before the first token. Where would you look?"


3. Choosing an API style: REST, gRPC, streaming

The question. How should a client and an AI service talk — and how do we send an answer word by word?

Start from nothing.

  • REST over HTTP + JSON — resources and verbs (POST /v1/answers). Human-readable, works everywhere. The default for public APIs.
  • gRPC — functions defined in a schema (Protocol Buffers), compact binary messages, running over HTTP/2 (many requests multiplexed over one connection). Common between internal services.
  • Streaming the answer. LLMs produce tokens one at a time. Instead of waiting for the full reply, servers stream:
    • Server-Sent Events (SSE) — one HTTP response that stays open and sends data: lines. Simple; used by most LLM APIs.
    • WebSockets — a two-way channel, for when the client also sends continuously (voice, live collaboration).
  • HTTP/3 runs HTTP over QUIC (on UDP), avoiding TCP's head-of-line blocking and making connection setup faster.
three horizontal timelines — "plain request" (one arrow back at the end), "SSE stream" (many small arrows back as tokens appear), "WebSocket" (arrows in both directions).
Diagram— three horizontal timelines — "plain request" (one arrow back at the end), "SSE stream" (many small arrows back as tokens appear), "WebSocket" (arrows in both directions).

Worked example — what an SSE token stream looks like on the wire:

python
import json
import time

def sse_stream(tokens):
    for tok in tokens:
        time.sleep(0.05)                  # the model generating this token
        yield f"data: {json.dumps({'token': tok})}\n\n"
    yield "data: [DONE]\n\n"

start = time.perf_counter()
answer = []
for i, event in enumerate(sse_stream(["A", " page", " table", " maps", " pages", "."])):
    payload = event.removeprefix("data: ").strip()
    if payload == "[DONE]":
        break
    if i == 0:
        print(f"first token after {(time.perf_counter() - start) * 1000:.0f} ms")   # ~50 ms
    answer.append(json.loads(payload)["token"])
print(f"full answer after {(time.perf_counter() - start) * 1000:.0f} ms: {''.join(answer)}")  # ~300 ms
# Streaming shows text after ~50 ms; waiting for the whole reply would show nothing for ~300 ms.

A minimal Atlas endpoint (FastAPI) — shown for shape; needs pip install fastapi uvicorn:

python
from fastapi import FastAPI
from fastapi.responses import StreamingResponse
from pydantic import BaseModel

app = FastAPI()

class Question(BaseModel):
    text: str

@app.post("/v1/ask")
async def ask(q: Question):
    async def stream():
        for token in ["Paging", " splits", " memory", " into", " pages", "."]:
            yield f"data: {token}\n\n"
    return StreamingResponse(stream(), media_type="text/event-stream")

@app.get("/healthz")
def health():
    return {"status": "ok"}
Choose When
REST + JSON Public APIs, browsers, simplicity
gRPC Internal service-to-service, strict schemas, high call volume
SSE Server → client token streaming
WebSocket Two-way real-time traffic

4. Load balancing, reverse proxies and connection pools

The question. One server can handle 200 requests at a time. Atlas gets 2,000. Now what?

Start from nothing. Run many servers and put a load balancer in front to spread requests (round-robin, least-connections, or by load). A reverse proxy (nginx, Envoy) sits in front of servers too and handles TLS, compression, routing by URL and health checks, so app servers only see plain, healthy traffic.

On the client side, opening a new TCP + TLS connection per request costs round trips. A connection pool keeps a few connections open and reuses them.

many clients → one load balancer → three app servers (one greyed out as "failed health check"); inside the client, a small pool of three reusable connection pipes.
Diagram— many clients → one load balancer → three app servers (one greyed out as "failed health check"); inside the client, a small pool of three reusable connection pipes.

Real-life analogy. A restaurant host seating guests at whichever waiter has the fewest tables, and skipping the waiter who's on break.

Worked example — least-connections balancing:

python
import random

random.seed(1)
servers = {"gpu-a": 0, "gpu-b": 0, "gpu-c": 0}      # active requests per server
healthy = {"gpu-a", "gpu-b", "gpu-c"}
healthy.discard("gpu-c")                            # failed its health check

def pick():
    return min((s for s in servers if s in healthy), key=lambda s: servers[s])

for request in range(10):
    s = pick()
    servers[s] += 1
    if random.random() < 0.5:                       # some requests finish quickly
        servers[s] -= 1
print(servers)    # gpu-c never receives traffic

In production. LLM servers are uneven — one long generation can hog a GPU — so smarter routing uses queue length or KV-cache usage, not just connection counts (D12).


5. Timeouts, retries and rate limiting

The question. A model call hangs for 60 seconds, or fails 1 time in 50. How do we stay responsive without making things worse?

Start from nothing.

  • Timeout — never wait forever; give up after a budget and report an error (or fall back).
  • Retry with exponential backoff and jitter — retry transient failures after 0.1 s, 0.2 s, 0.4 s… plus randomness, so thousands of clients don't retry in sync and flatten the server (thundering herd). Only retry operations that are safe to repeat (idempotent), or send an idempotency key.
  • Rate limiting — cap how fast each user can call you. The classic token bucket: a bucket holds up to B tokens and refills at R per second; each request spends one token; an empty bucket means 429 Too Many Requests.
left, a timeline of retries at growing, slightly random gaps; right, a bucket with tokens dripping in from the top at a steady rate and requests taking tokens out of the bottom.
Diagram— left, a timeline of retries at growing, slightly random gaps; right, a bucket with tokens dripping in from the top at a steady rate and requests taking tokens out of the bottom.

Real-life analogy. Calling a busy shop: wait a bit longer before each redial (backoff), and don't all redial at the same second (jitter). A club bouncer letting in a few people per minute (rate limit).

Worked example.

python
import random
import time

random.seed(7)

def flaky_model_call(attempt):
    if attempt < 2:
        raise TimeoutError("model busy")
    return "answer"

def call_with_retries(max_attempts=5, base=0.05, cap=1.0):
    for attempt in range(max_attempts):
        try:
            return flaky_model_call(attempt)
        except TimeoutError as err:
            delay = random.uniform(0, min(cap, base * 2 ** attempt))   # "full jitter"
            print(f"attempt {attempt + 1} failed ({err}); retrying in {delay:.3f}s")
            time.sleep(delay)
    raise RuntimeError("gave up")

print(call_with_retries())

class TokenBucket:
    def __init__(self, rate_per_s, burst):
        self.rate, self.capacity = rate_per_s, burst
        self.tokens, self.last = float(burst), time.monotonic()

    def allow(self):
        now = time.monotonic()
        self.tokens = min(self.capacity, self.tokens + (now - self.last) * self.rate)
        self.last = now
        if self.tokens >= 1:
            self.tokens -= 1
            return True
        return False

bucket = TokenBucket(rate_per_s=5, burst=3)
print(["ok" if bucket.allow() else "429" for _ in range(6)])   # 3 ok, then 429s
time.sleep(0.4)
print("after 0.4s:", "ok" if bucket.allow() else "429")         # refilled ~2 tokens

Common traps

  • Retrying non-idempotent calls (charging a card twice).
  • Timeouts longer than the caller's timeout — work continues after nobody is waiting.
  • Retrying on 429 without waiting for the server's Retry-After.

In production / interviews. Interview question: "Design rate limiting for an LLM API priced per token, where one request can be 10 tokens or 100,000." (Hint: limit tokens per minute, not just requests.)

From production

A model API sometimes takes 30 seconds under load, and clients retry immediately on timeout, all at once.

What client-side change prevents a retry storm?

  1. Disable timeouts
  2. Shorter timeouts with immediate retries
  3. Retries with exponential backoff and jitter, a retry budget, and idempotency keys so repeated requests don't duplicate work
  4. More client threads
Show the answer

C. Retries with exponential backoff and jitter, a retry budget, and idempotency keys so repeated requests don't duplicate work — Synchronized immediate retries multiply load on an already struggling server; backoff with jitter spreads them out and budgets cap them.


Check yourself

  1. Recall: Which layer guarantees packets arrive in order — IP, TCP or HTTP?
  2. Apply: Should a chat UI wait for the full answer or stream it? Which mechanism?
  3. Predict: A bucket with burst 3 and rate 5/s receives 6 instant requests. How many succeed?
Answers
  1. TCP.
  2. Stream it, usually with Server-Sent Events — it cuts time to first token.
  3. 3 (the burst); the rest get 429 until tokens refill.

Exercise

Write resilient_call(fn, timeout_s, max_attempts) that runs fn in a thread, gives up on any attempt longer than timeout_s, retries with full-jitter backoff, and never exceeds a total budget of timeout_s × max_attempts. Test it with a function that sleeps randomly between 0 and 0.3 s.

Solution
python
import random
import time
from concurrent.futures import ThreadPoolExecutor, TimeoutError as FutureTimeout

def resilient_call(fn, timeout_s=0.15, max_attempts=4, base=0.02):
    deadline = time.monotonic() + timeout_s * max_attempts
    with ThreadPoolExecutor(max_workers=max_attempts) as pool:
        for attempt in range(max_attempts):
            remaining = deadline - time.monotonic()
            if remaining <= 0:
                break
            future = pool.submit(fn)
            try:
                return future.result(timeout=min(timeout_s, remaining))
            except FutureTimeout:
                time.sleep(random.uniform(0, base * 2 ** attempt))
    raise TimeoutError("all attempts timed out")

random.seed(3)
def slow_model():
    time.sleep(random.uniform(0, 0.3))
    return "answer"

try:
    print(resilient_call(slow_model))
except TimeoutError as e:
    print(e)

This unlocks:

  • Databases — where Atlas stores what the request needs.
  • D11 Model deployment and D12 Inference fundamentals — streaming, routing, time to first token.
  • D14 Distributed systems — retries, idempotency and failure as normal events.

Sources: Kurose & Ross, Computer Networking: A Top-Down Approach; RFC 8446 (TLS 1.3), RFC 9114 (HTTP/3); WHATWG HTML spec — Server-sent events; AWS Architecture Blog, "Exponential Backoff and Jitter"; gRPC docs.