VFlash: Fast, Accurate Visual Planning via Flash Primitive Decoding
u_Unrolled
D1: CS & Systems Foundations · Interview Patterns and Capstone — An Async, Rate-Limited LLM Batch Client
Phase 1/Domain 1/Lesson 18

Interview Patterns and Capstone — An Async, Rate-Limited LLM Batch Client

15 min read·Foundational·Free lesson
Interview Patterns and Capstone — An Async, Rate-Limited LLM Batch Client

The scenario. In an AI engineering interview loop, there's usually a Python round: one or two algorithm-style problems, then a practical task ("build a client that calls this API 10,000 times without getting rate-limited"). Interviewers watch how you think: do you ask about input size, pick the right data structure, handle failures, and test your code? This lesson gives you the patterns, then a capstone that uses nearly everything in the path.

You already know: the whole Python path. Each pattern below points back to the lesson it comes from.


1. Six patterns that solve most Python interview problems

The question. Is there a small set of ideas behind most coding questions?

Start from nothing. Yes. Before coding, say the brute force and its complexity, then look for which pattern removes the repeated work:

Pattern Use when Typical cost From
Hash map / set "seen before?", counting, pairing O(n) Dicts and sets
Two pointers sorted arrays, pairs, in-place cleanup O(n) Lists and tuples
Sliding window best/longest contiguous run O(n) Lists, deque
Heap top k, merging sorted streams O(n log k) Data structures
Stack / queue nesting, undo, BFS O(n) Lists, deque
Binary search sorted data, "smallest x such that…" O(log n) Data structures

Three of them in action:

python
from collections import Counter
import heapq

def two_sum(nums, target):                  # hash map: one pass instead of O(n²) pairs
    seen = {}
    for i, n in enumerate(nums):
        if target - n in seen:
            return seen[target - n], i
        seen[n] = i
    return None

def longest_unique_run(s):                  # sliding window over a string
    last, start, best = {}, 0, 0
    for i, ch in enumerate(s):
        if ch in last and last[ch] >= start:
            start = last[ch] + 1            # move the window past the repeat
        last[ch] = i
        best = max(best, i - start + 1)
    return best

def top_k_words(words, k):                  # heap of size k: O(n log k)
    counts = Counter(words)
    return heapq.nlargest(k, counts.items(), key=lambda kv: kv[1])

print(two_sum([3, 8, 11, 4], 12))
print(longest_unique_run("abcabcbb"), longest_unique_run("pwwkew"))
print(top_k_words("the cat and the hat and the bat".split(), 2))

This prints (1, 3), then 3 3, then [('the', 3), ('and', 2)].

six cues mapped to patterns and costs: "seen before? / count / find a pair" → hash map or set, O(n); "sorted array / pairs" → two pointers, O(n); "best contiguous run" → sliding window, O(n); "top k / merge streams" → heap, O(n log k); "nesting / undo / BFS" → stack or queue, O(n); "sorted / smallest x such that" → binary search, O(log n).
Diagram— six cues mapped to patterns and costs: "seen before? / count / find a pair" → hash map or set, O(n); "sorted array / pairs" → two pointers, O(n); "best contiguous run" → sliding window, O(n); "top k / merge streams" → heap, O(n log k); "nesting / undo / BFS" → stack or queue, O(n); "sorted / smallest x such that" → binary search, O(log n).

These aren't just interview tricks. The same six patterns run inside production AI systems you'll study later. A sliding window bounds how much chat history fits into a model's context. A heap picks the top-k retrieved documents. Binary search over sorted token offsets maps a character position back to a token. BFS over a dependency graph decides which pipeline steps can run in parallel. Interviewers use these problems because they're a compact way to test whether you see the repeated work and know a structure that removes it.

Key idea. In interviews, narrate: "Brute force is O(n²) because…; a dict removes the inner loop, giving O(n) time and O(n) space." Correct code without a stated complexity loses points; a clear explanation of a slightly slower solution often wins them.

From production

Interview question: 'Given 10 million log lines streaming in, return the 100 most frequent user ids at any time.' A candidate proposes sorting all counts every time.

What's the better design?

  1. Use a set of user ids
  2. Re-read the whole log for each request
  3. Keep a Counter (dict) of counts as lines arrive, and use heapq.nlargest(100, ...) on request: O(u log 100) where u is unique users
  4. Store all lines in a list and sort on request
Show the answer

C. Keep a Counter (dict) of counts as lines arrive, and use heapq.nlargest(100, ...) on request: O(u log 100) where u is unique users — Counting is O(1) per line with a dict. Taking the top 100 with a size-100 heap avoids a full sort of all unique users. Mention the trade-off: approximate structures (Count-Min Sketch) if memory becomes the limit.

2. Python-specific questions interviewers love

The question. What do interviewers ask to check that you really know Python?

Start from nothing. These come up again and again, and every one was a section of this path:

  • "What's wrong with def f(x, items=[])?" The default is created once and shared (Functions).
  • "Is a list or a set faster for in?" Set: O(1) hashing vs an O(n) scan (Dicts and sets).
  • "Shallow vs deep copy?" copy() shares nested objects; deepcopy doesn't (Lists and tuples).
  • "Threads or processes?" Threads/async for I/O, processes for CPU-bound pure Python, because of the GIL (The GIL).
  • "How would you process a 50 GB file?" Stream it with a generator, never readlines() (Iterators and generators).
  • "How do you guarantee cleanup?" A context manager / with (Decorators and context managers).
  • "is vs ==?" Identity vs equality (Values, names and types).
seven interview questions each linked to the lesson that answers it: items=[] → Functions; list or set for in → Dicts and sets; shallow vs deep copy → Lists and tuples; threads or processes → The GIL; a 50 GB file → Iteration; guaranteed cleanup → Context managers; is vs == → Values and names.
Diagram— seven interview questions each linked to the lesson that answers it: items=[] → Functions; list or set for in → Dicts and sets; shallow vs deep copy → Lists and tuples; threads or processes → The GIL; a 50 GB file → Iteration; guaranteed cleanup → Context managers; is vs == → Values and names.

How to answer them well. Give the mechanism, then a consequence from real code. "The default list is created once, when def runs, so every call shares it. I've seen it leak one user's chat history into another's" is a stronger answer than "never use mutable defaults". Interviewers for AI engineering roles often follow up with a production twist: "and how would you find this bug in a live service?" Each lesson in this path ended with exactly those production failure modes. Use them.

Quick check

An interviewer asks: 'Your function receives a generator. You call len() on it to log the size. What happens?'

  1. It returns 0
  2. TypeError: generators have no len(); and if you iterate to count, the generator is consumed
  3. It returns None
  4. It returns the number of items
Show the answer

B. TypeError: generators have no len(); and if you iterate to count, the generator is consumed — Generators don't know their length in advance, so len() raises TypeError. Counting by iterating exhausts it, leaving nothing for the real work. Count while processing instead.

3. Capstone: the batch client

The question. Can you build the tool the whole path has been pointing at?

Start from nothing. The task: grade a batch of student answers with an LLM API. Requirements:

  1. Typed requests and results (dataclasses, an Enum for status). Classes and types.
  2. At most N requests in flight (semaphore) and R requests per second (token bucket). Async.
  3. Timeouts per call and retries with backoff and jitter for transient errors only. Decorators, async.
  4. Failures recorded, never fatal: one bad item doesn't lose the batch. Errors.
  5. Results in input order, plus a summary: successes, failures, retries, p95 latency. NumPy-style stats, profiling.
  6. Swappable provider through a Protocol, so tests use a fake. Protocols, testing.
the client's pipeline: items → token bucket → semaphore → provider call with timeout → retry loop on transient errors → ordered results and a summary report.
Diagram— the client's pipeline: items → token bucket → semaphore → provider call with timeout → retry loop on transient errors → ordered results and a summary report.
python
import asyncio, random, time
from dataclasses import dataclass, field
from enum import Enum
from typing import Protocol, Optional

# ---- types -----------------------------------------------------------------
class Status(Enum):
    OK = "ok"
    FAILED = "failed"

@dataclass(frozen=True)
class GradeRequest:
    id: int
    answer: str

@dataclass
class GradeResult:
    id: int
    status: Status
    score: Optional[int] = None
    error: str = ""
    attempts: int = 0
    latency: float = 0.0

class Transient(Exception):
    """429, 503, timeouts: worth retrying."""

class Permanent(Exception):
    """400-style errors: retrying won't help."""

class Provider(Protocol):
    async def grade(self, answer: str) -> int: ...

# ---- rate limiting -----------------------------------------------------------
class TokenBucket:
    def __init__(self, rate: float, capacity: int):
        self.rate, self.capacity, self.tokens = rate, capacity, float(capacity)
        self.updated = time.monotonic()
        self.lock = asyncio.Lock()

    async def acquire(self) -> None:
        async with self.lock:
            while True:
                now = time.monotonic()
                self.tokens = min(self.capacity, self.tokens + (now - self.updated) * self.rate)
                self.updated = now
                if self.tokens >= 1:
                    self.tokens -= 1
                    return
                await asyncio.sleep((1 - self.tokens) / self.rate)

# ---- the client ----------------------------------------------------------------
@dataclass
class BatchClient:
    provider: Provider
    concurrency: int = 8
    rate_per_s: float = 50
    timeout_s: float = 0.5
    max_attempts: int = 4
    rng: random.Random = field(default_factory=lambda: random.Random(0))

    async def _one(self, req: GradeRequest, sem: asyncio.Semaphore, bucket: TokenBucket) -> GradeResult:
        start = time.perf_counter()
        for attempt in range(1, self.max_attempts + 1):
            await bucket.acquire()                       # requests per second
            try:
                async with sem:                          # requests in flight
                    score = await asyncio.wait_for(self.provider.grade(req.answer), self.timeout_s)
                return GradeResult(req.id, Status.OK, score, attempts=attempt,
                                   latency=time.perf_counter() - start)
            except Permanent as err:
                return GradeResult(req.id, Status.FAILED, error=f"permanent: {err}", attempts=attempt,
                                   latency=time.perf_counter() - start)
            except (Transient, asyncio.TimeoutError) as err:
                if attempt == self.max_attempts:
                    return GradeResult(req.id, Status.FAILED, error=f"gave up: {type(err).__name__}",
                                       attempts=attempt, latency=time.perf_counter() - start)
                await asyncio.sleep(0.02 * 2 ** attempt * self.rng.uniform(0.5, 1.5))

    async def run(self, requests: list) -> list:
        sem = asyncio.Semaphore(self.concurrency)
        bucket = TokenBucket(self.rate_per_s, capacity=self.concurrency)
        return await asyncio.gather(*(self._one(r, sem, bucket) for r in requests))

def summarise(results: list) -> dict:
    ok = [r for r in results if r.status is Status.OK]
    latencies = sorted(r.latency for r in ok)
    p95 = latencies[int(0.95 * (len(latencies) - 1))] if latencies else 0.0
    return {
        "ok": len(ok),
        "failed": len(results) - len(ok),
        "retries": sum(r.attempts - 1 for r in results),
        "p95_s": round(p95, 2),
    }

# ---- a fake provider for testing: flaky, sometimes slow, sometimes permanently bad ----
class FakeGrader:
    def __init__(self, seed: int):
        self.rng = random.Random(seed)
        self.calls = 0
        self.in_flight = self.peak = 0

    async def grade(self, answer: str) -> int:
        self.calls += 1
        self.in_flight += 1
        self.peak = max(self.peak, self.in_flight)
        try:
            if not answer.strip():
                raise Permanent("empty answer")
            roll = self.rng.random()
            if roll < 0.15:
                raise Transient("429 too many requests")
            await asyncio.sleep(2.0 if roll < 0.20 else 0.05)   # 5% of calls hang
            return min(10, len(answer.split()))
        finally:
            self.in_flight -= 1

async def main():
    answers = [("the GIL " * (i % 6)).strip() or "" for i in range(60)]
    requests = [GradeRequest(i, a) for i, a in enumerate(answers)]
    grader = FakeGrader(seed=5)
    client = BatchClient(grader, concurrency=8, rate_per_s=50)

    start = time.perf_counter()
    results = await client.run(requests)
    elapsed = time.perf_counter() - start

    assert [r.id for r in results] == list(range(60))          # input order kept
    assert grader.peak <= 8                                     # concurrency respected
    print(summarise(results), f"calls={grader.calls}", f"peak={grader.peak}", f"{elapsed:.1f}s")
    print([r.error for r in results if r.status is Status.FAILED][:3])

asyncio.run(main())

Read the output against the requirements:

  • Every 6th answer is empty (i % 6 == 0), so those 10 items fail permanently on the first attempt: no wasted retries.
  • 429s and hung calls (cut off at 0.5 s) are retried with backoff, so the retries count is well above zero, and in this run all 50 valid answers still end up ok.
  • The asserts prove results are in input order and that no more than 8 calls were ever in flight.
  • At 50 requests per second, the 60 items plus retries take a couple of seconds.

From production

In the capstone client, a teammate moves await bucket.acquire() outside the retry loop, so it's called once per item instead of once per attempt.

What's the consequence?

  1. Retries bypass the rate limit, so a burst of 429s triggers a burst of un-throttled retries: exactly when the provider is struggling
  2. Results come back out of order
  3. The semaphore stops working
  4. Nothing changes
Show the answer

A. Retries bypass the rate limit, so a burst of 429s triggers a burst of un-throttled retries: exactly when the provider is struggling — Every attempt is a real request that counts against the provider's limit. Retries must go through the limiter too; otherwise the client hammers the API hardest exactly when it's already rate-limiting you.

4. Taking it further

The question. What turns this into a portfolio project?

Start from nothing. Each step uses a lesson from the path:

  • Swap FakeGrader for a real provider using httpx.AsyncClient (one shared client, so connections are reused).
  • Checkpoint results to a JSONL file as they finish, and skip completed ids on restart, so a crash at item 9,000 doesn't redo 9,000 calls.
  • Respect the provider's Retry-After header on 429s instead of guessing the backoff.
  • Add a tokens-per-minute bucket alongside requests-per-second.
  • Package it with uv, add pytest tests that use FakeGrader (including the order and peak assertions above), and type-check with mypy.
the batch grading client in the centre, surrounded by five upgrades: a real provider through one shared httpx.AsyncClient, checkpoint + resume with JSONL, Retry-After on 429, a tokens-per-minute bucket, and uv + pytest + mypy with the fake grader.
Diagram— the batch grading client in the centre, surrounded by five upgrades: a real provider through one shared httpx.AsyncClient, checkpoint + resume with JSONL, Retry-After on 429, a tokens-per-minute bucket, and uv + pytest + mypy with the fake grader.

The checkpoint upgrade is the one that saves real money, so here is its core: append each result as a JSON line the moment it finishes, and on start-up, skip every ID already in the file.

python
import json, os, tempfile

path = os.path.join(tempfile.mkdtemp(), "results.jsonl")

def done_ids(path):
    if not os.path.exists(path):
        return set()
    with open(path, encoding="utf-8") as f:
        return {json.loads(line)["id"] for line in f if line.strip()}

def run(items, crash_at=None):
    finished = done_ids(path)
    calls = 0
    with open(path, "a", encoding="utf-8") as out:
        for item in items:
            if item in finished:
                continue                                   # already graded in an earlier run
            if item == crash_at:
                raise RuntimeError(f"worker died at item {item}")
            calls += 1                                     # stands in for one paid API call
            out.write(json.dumps({"id": item, "score": item % 5}) + "\n")
            out.flush()                                    # the line is on disk before we move on
    return calls

items = list(range(10))
try:
    run(items, crash_at=7)
except RuntimeError as e:
    print(e, "| saved so far:", len(done_ids(path)))
print("calls on restart:", run(items), "| total saved:", len(done_ids(path)))

The first run dies at item 7 with 7 results already safely on disk. The restart makes only 3 calls and ends with all 10 saved: nothing is graded (or paid for) twice. Two details make it robust. flush() after every line means a crash loses at most the item in progress. A set of finished IDs makes the skip check O(1) per item (Dicts and sets).

In production. This is the "idempotent, resumable job" pattern behind evaluation harnesses, backfills and bulk embedding jobs. At larger scale, the JSONL file becomes a database table or object-store files keyed by item ID, and the skip check becomes a query. The principle is the same: write results as you go, make re-running safe, and never let a crash cost more than the work in flight.

From production

Your batch grader writes all results to a JSON file only after the last of 20,000 items finishes. At item 18,500 the process is killed by a deploy.

What's lost, and what design prevents it?

  1. All 18,500 paid calls' results are lost; append each result as it completes (e.g. JSONL with flush) and skip finished IDs on restart
  2. Nothing; Python saves partial results automatically
  3. Only the last item
  4. The results are in the OS cache and can be recovered
Show the answer

A. All 18,500 paid calls' results are lost; append each result as it completes (e.g. JSONL with flush) and skip finished IDs on restart — Results held only in memory die with the process. Writing each result as it finishes, plus a resume step that skips completed IDs, turns a crash into a cheap restart instead of a full re-run.


Check yourself

  1. Which pattern turns "find two numbers that sum to target" from O(n²) into O(n)?
  2. Why must retries go through the rate limiter?
  3. Why are permanent errors not retried?
  4. How does the capstone keep results in input order even though calls finish in random order?
Answers
  1. A hash map of values already seen: for each number, check whether target - n is in the dict.
  2. Each retry is a real request that counts against the limit; unthrottled retries make rate limiting worse.
  3. They'll fail the same way every time (for example, an empty answer), so retrying only wastes time, money and quota.
  4. asyncio.gather returns results in the order of the awaitables passed in, regardless of completion order.

Exercise

Add checkpointing to the capstone: run accepts a done dict of id -> GradeResult from a previous run, skips those ids, and returns the merged results in order. Test it by running the batch, deleting half the results, and re-running: the second run should only call the provider for the missing half.

Solution
python
import asyncio

async def run_with_checkpoint(client_run, requests, done):
    todo = [r for r in requests if r.id not in done]
    fresh = await client_run(todo)
    merged = dict(done)
    merged.update({res["id"]: res for res in fresh})
    return [merged[r.id] for r in requests]          # back in input order

class Req:
    def __init__(self, i):
        self.id = i

async def fake_client_run(reqs):
    fake_client_run.calls += len(reqs)
    await asyncio.sleep(0.01)
    return [{"id": r.id, "score": r.id % 10} for r in reqs]
fake_client_run.calls = 0

async def main():
    requests = [Req(i) for i in range(20)]
    first = await run_with_checkpoint(fake_client_run, requests, done={})
    half = {res["id"]: res for res in first if res["id"] % 2 == 0}   # pretend the odd half was lost
    calls_before = fake_client_run.calls
    second = await run_with_checkpoint(fake_client_run, requests, done=half)
    print("provider calls on re-run:", fake_client_run.calls - calls_before)
    assert second == first
    print("results identical and in order")

asyncio.run(main())

The re-run calls the provider only 10 times, for the missing ids. In the real client, done comes from reading the JSONL checkpoint file at start-up, and each result is appended to that file the moment it finishes.


You've finished the core Python path. From here:

  • PyTorch engineering — tensors, autograd and training loops, built on NumPy and the performance ideas here.
  • High-throughput networking — what's under await: sockets, connection pools, backpressure and retries at scale.