The scenario. In an AI engineering interview loop, there's usually a Python round: one or two algorithm-style problems, then a practical task ("build a client that calls this API 10,000 times without getting rate-limited"). Interviewers watch how you think: do you ask about input size, pick the right data structure, handle failures, and test your code? This lesson gives you the patterns, then a capstone that uses nearly everything in the path.
You already know: the whole Python path. Each pattern below points back to the lesson it comes from.
1. Six patterns that solve most Python interview problems
The question. Is there a small set of ideas behind most coding questions?
Start from nothing. Yes. Before coding, say the brute force and its complexity, then look for which pattern removes the repeated work:
| Pattern | Use when | Typical cost | From |
|---|---|---|---|
| Hash map / set | "seen before?", counting, pairing | O(n) | Dicts and sets |
| Two pointers | sorted arrays, pairs, in-place cleanup | O(n) | Lists and tuples |
| Sliding window | best/longest contiguous run | O(n) | Lists, deque |
| Heap | top k, merging sorted streams | O(n log k) | Data structures |
| Stack / queue | nesting, undo, BFS | O(n) | Lists, deque |
| Binary search | sorted data, "smallest x such that…" | O(log n) | Data structures |
Three of them in action:
from collections import Counter
import heapq
def two_sum(nums, target): # hash map: one pass instead of O(n²) pairs
seen = {}
for i, n in enumerate(nums):
if target - n in seen:
return seen[target - n], i
seen[n] = i
return None
def longest_unique_run(s): # sliding window over a string
last, start, best = {}, 0, 0
for i, ch in enumerate(s):
if ch in last and last[ch] >= start:
start = last[ch] + 1 # move the window past the repeat
last[ch] = i
best = max(best, i - start + 1)
return best
def top_k_words(words, k): # heap of size k: O(n log k)
counts = Counter(words)
return heapq.nlargest(k, counts.items(), key=lambda kv: kv[1])
print(two_sum([3, 8, 11, 4], 12))
print(longest_unique_run("abcabcbb"), longest_unique_run("pwwkew"))
print(top_k_words("the cat and the hat and the bat".split(), 2))
This prints (1, 3), then 3 3, then [('the', 3), ('and', 2)].
These aren't just interview tricks. The same six patterns run inside production AI systems you'll study later. A sliding window bounds how much chat history fits into a model's context. A heap picks the top-k retrieved documents. Binary search over sorted token offsets maps a character position back to a token. BFS over a dependency graph decides which pipeline steps can run in parallel. Interviewers use these problems because they're a compact way to test whether you see the repeated work and know a structure that removes it.
Key idea. In interviews, narrate: "Brute force is O(n²) because…; a dict removes the inner loop, giving O(n) time and O(n) space." Correct code without a stated complexity loses points; a clear explanation of a slightly slower solution often wins them.
From production
Interview question: 'Given 10 million log lines streaming in, return the 100 most frequent user ids at any time.' A candidate proposes sorting all counts every time.
What's the better design?
Show the answer
C. Keep a Counter (dict) of counts as lines arrive, and use heapq.nlargest(100, ...) on request: O(u log 100) where u is unique users — Counting is O(1) per line with a dict. Taking the top 100 with a size-100 heap avoids a full sort of all unique users. Mention the trade-off: approximate structures (Count-Min Sketch) if memory becomes the limit.
2. Python-specific questions interviewers love
The question. What do interviewers ask to check that you really know Python?
Start from nothing. These come up again and again, and every one was a section of this path:
- "What's wrong with
def f(x, items=[])?" The default is created once and shared (Functions). - "Is a list or a set faster for
in?" Set: O(1) hashing vs an O(n) scan (Dicts and sets). - "Shallow vs deep copy?"
copy()shares nested objects;deepcopydoesn't (Lists and tuples). - "Threads or processes?" Threads/async for I/O, processes for CPU-bound pure Python, because of the GIL (The GIL).
- "How would you process a 50 GB file?" Stream it with a generator, never
readlines()(Iterators and generators). - "How do you guarantee cleanup?" A context manager /
with(Decorators and context managers). - "
isvs==?" Identity vs equality (Values, names and types).
How to answer them well. Give the mechanism, then a consequence from real code. "The default list is created once, when def runs, so every call shares it. I've seen it leak one user's chat history into another's" is a stronger answer than "never use mutable defaults". Interviewers for AI engineering roles often follow up with a production twist: "and how would you find this bug in a live service?" Each lesson in this path ended with exactly those production failure modes. Use them.
Quick check
An interviewer asks: 'Your function receives a generator. You call len() on it to log the size. What happens?'
Show the answer
B. TypeError: generators have no len(); and if you iterate to count, the generator is consumed — Generators don't know their length in advance, so len() raises TypeError. Counting by iterating exhausts it, leaving nothing for the real work. Count while processing instead.
3. Capstone: the batch client
The question. Can you build the tool the whole path has been pointing at?
Start from nothing. The task: grade a batch of student answers with an LLM API. Requirements:
- Typed requests and results (dataclasses, an Enum for status). Classes and types.
- At most N requests in flight (semaphore) and R requests per second (token bucket). Async.
- Timeouts per call and retries with backoff and jitter for transient errors only. Decorators, async.
- Failures recorded, never fatal: one bad item doesn't lose the batch. Errors.
- Results in input order, plus a summary: successes, failures, retries, p95 latency. NumPy-style stats, profiling.
- Swappable provider through a Protocol, so tests use a fake. Protocols, testing.
import asyncio, random, time
from dataclasses import dataclass, field
from enum import Enum
from typing import Protocol, Optional
# ---- types -----------------------------------------------------------------
class Status(Enum):
OK = "ok"
FAILED = "failed"
@dataclass(frozen=True)
class GradeRequest:
id: int
answer: str
@dataclass
class GradeResult:
id: int
status: Status
score: Optional[int] = None
error: str = ""
attempts: int = 0
latency: float = 0.0
class Transient(Exception):
"""429, 503, timeouts: worth retrying."""
class Permanent(Exception):
"""400-style errors: retrying won't help."""
class Provider(Protocol):
async def grade(self, answer: str) -> int: ...
# ---- rate limiting -----------------------------------------------------------
class TokenBucket:
def __init__(self, rate: float, capacity: int):
self.rate, self.capacity, self.tokens = rate, capacity, float(capacity)
self.updated = time.monotonic()
self.lock = asyncio.Lock()
async def acquire(self) -> None:
async with self.lock:
while True:
now = time.monotonic()
self.tokens = min(self.capacity, self.tokens + (now - self.updated) * self.rate)
self.updated = now
if self.tokens >= 1:
self.tokens -= 1
return
await asyncio.sleep((1 - self.tokens) / self.rate)
# ---- the client ----------------------------------------------------------------
@dataclass
class BatchClient:
provider: Provider
concurrency: int = 8
rate_per_s: float = 50
timeout_s: float = 0.5
max_attempts: int = 4
rng: random.Random = field(default_factory=lambda: random.Random(0))
async def _one(self, req: GradeRequest, sem: asyncio.Semaphore, bucket: TokenBucket) -> GradeResult:
start = time.perf_counter()
for attempt in range(1, self.max_attempts + 1):
await bucket.acquire() # requests per second
try:
async with sem: # requests in flight
score = await asyncio.wait_for(self.provider.grade(req.answer), self.timeout_s)
return GradeResult(req.id, Status.OK, score, attempts=attempt,
latency=time.perf_counter() - start)
except Permanent as err:
return GradeResult(req.id, Status.FAILED, error=f"permanent: {err}", attempts=attempt,
latency=time.perf_counter() - start)
except (Transient, asyncio.TimeoutError) as err:
if attempt == self.max_attempts:
return GradeResult(req.id, Status.FAILED, error=f"gave up: {type(err).__name__}",
attempts=attempt, latency=time.perf_counter() - start)
await asyncio.sleep(0.02 * 2 ** attempt * self.rng.uniform(0.5, 1.5))
async def run(self, requests: list) -> list:
sem = asyncio.Semaphore(self.concurrency)
bucket = TokenBucket(self.rate_per_s, capacity=self.concurrency)
return await asyncio.gather(*(self._one(r, sem, bucket) for r in requests))
def summarise(results: list) -> dict:
ok = [r for r in results if r.status is Status.OK]
latencies = sorted(r.latency for r in ok)
p95 = latencies[int(0.95 * (len(latencies) - 1))] if latencies else 0.0
return {
"ok": len(ok),
"failed": len(results) - len(ok),
"retries": sum(r.attempts - 1 for r in results),
"p95_s": round(p95, 2),
}
# ---- a fake provider for testing: flaky, sometimes slow, sometimes permanently bad ----
class FakeGrader:
def __init__(self, seed: int):
self.rng = random.Random(seed)
self.calls = 0
self.in_flight = self.peak = 0
async def grade(self, answer: str) -> int:
self.calls += 1
self.in_flight += 1
self.peak = max(self.peak, self.in_flight)
try:
if not answer.strip():
raise Permanent("empty answer")
roll = self.rng.random()
if roll < 0.15:
raise Transient("429 too many requests")
await asyncio.sleep(2.0 if roll < 0.20 else 0.05) # 5% of calls hang
return min(10, len(answer.split()))
finally:
self.in_flight -= 1
async def main():
answers = [("the GIL " * (i % 6)).strip() or "" for i in range(60)]
requests = [GradeRequest(i, a) for i, a in enumerate(answers)]
grader = FakeGrader(seed=5)
client = BatchClient(grader, concurrency=8, rate_per_s=50)
start = time.perf_counter()
results = await client.run(requests)
elapsed = time.perf_counter() - start
assert [r.id for r in results] == list(range(60)) # input order kept
assert grader.peak <= 8 # concurrency respected
print(summarise(results), f"calls={grader.calls}", f"peak={grader.peak}", f"{elapsed:.1f}s")
print([r.error for r in results if r.status is Status.FAILED][:3])
asyncio.run(main())
Read the output against the requirements:
- Every 6th answer is empty (
i % 6 == 0), so those 10 items fail permanently on the first attempt: no wasted retries. - 429s and hung calls (cut off at 0.5 s) are retried with backoff, so the
retriescount is well above zero, and in this run all 50 valid answers still end upok. - The asserts prove results are in input order and that no more than 8 calls were ever in flight.
- At 50 requests per second, the 60 items plus retries take a couple of seconds.
From production
In the capstone client, a teammate moves await bucket.acquire() outside the retry loop, so it's called once per item instead of once per attempt.
What's the consequence?
Show the answer
A. Retries bypass the rate limit, so a burst of 429s triggers a burst of un-throttled retries: exactly when the provider is struggling — Every attempt is a real request that counts against the provider's limit. Retries must go through the limiter too; otherwise the client hammers the API hardest exactly when it's already rate-limiting you.
4. Taking it further
The question. What turns this into a portfolio project?
Start from nothing. Each step uses a lesson from the path:
- Swap
FakeGraderfor a real provider usinghttpx.AsyncClient(one shared client, so connections are reused). - Checkpoint results to a JSONL file as they finish, and skip completed ids on restart, so a crash at item 9,000 doesn't redo 9,000 calls.
- Respect the provider's
Retry-Afterheader on 429s instead of guessing the backoff. - Add a tokens-per-minute bucket alongside requests-per-second.
- Package it with
uv, addpytesttests that useFakeGrader(including the order and peak assertions above), and type-check with mypy.
The checkpoint upgrade is the one that saves real money, so here is its core: append each result as a JSON line the moment it finishes, and on start-up, skip every ID already in the file.
import json, os, tempfile
path = os.path.join(tempfile.mkdtemp(), "results.jsonl")
def done_ids(path):
if not os.path.exists(path):
return set()
with open(path, encoding="utf-8") as f:
return {json.loads(line)["id"] for line in f if line.strip()}
def run(items, crash_at=None):
finished = done_ids(path)
calls = 0
with open(path, "a", encoding="utf-8") as out:
for item in items:
if item in finished:
continue # already graded in an earlier run
if item == crash_at:
raise RuntimeError(f"worker died at item {item}")
calls += 1 # stands in for one paid API call
out.write(json.dumps({"id": item, "score": item % 5}) + "\n")
out.flush() # the line is on disk before we move on
return calls
items = list(range(10))
try:
run(items, crash_at=7)
except RuntimeError as e:
print(e, "| saved so far:", len(done_ids(path)))
print("calls on restart:", run(items), "| total saved:", len(done_ids(path)))
The first run dies at item 7 with 7 results already safely on disk. The restart makes only 3 calls and ends with all 10 saved: nothing is graded (or paid for) twice. Two details make it robust. flush() after every line means a crash loses at most the item in progress. A set of finished IDs makes the skip check O(1) per item (Dicts and sets).
In production. This is the "idempotent, resumable job" pattern behind evaluation harnesses, backfills and bulk embedding jobs. At larger scale, the JSONL file becomes a database table or object-store files keyed by item ID, and the skip check becomes a query. The principle is the same: write results as you go, make re-running safe, and never let a crash cost more than the work in flight.
From production
Your batch grader writes all results to a JSON file only after the last of 20,000 items finishes. At item 18,500 the process is killed by a deploy.
What's lost, and what design prevents it?
Show the answer
A. All 18,500 paid calls' results are lost; append each result as it completes (e.g. JSONL with flush) and skip finished IDs on restart — Results held only in memory die with the process. Writing each result as it finishes, plus a resume step that skips completed IDs, turns a crash into a cheap restart instead of a full re-run.
Check yourself
- Which pattern turns "find two numbers that sum to target" from O(n²) into O(n)?
- Why must retries go through the rate limiter?
- Why are permanent errors not retried?
- How does the capstone keep results in input order even though calls finish in random order?
Answers
- A hash map of values already seen: for each number, check whether
target - nis in the dict. - Each retry is a real request that counts against the limit; unthrottled retries make rate limiting worse.
- They'll fail the same way every time (for example, an empty answer), so retrying only wastes time, money and quota.
asyncio.gatherreturns results in the order of the awaitables passed in, regardless of completion order.
Exercise
Add checkpointing to the capstone: run accepts a done dict of id -> GradeResult from a previous run, skips those ids, and returns the merged results in order. Test it by running the batch, deleting half the results, and re-running: the second run should only call the provider for the missing half.
Solution
import asyncio
async def run_with_checkpoint(client_run, requests, done):
todo = [r for r in requests if r.id not in done]
fresh = await client_run(todo)
merged = dict(done)
merged.update({res["id"]: res for res in fresh})
return [merged[r.id] for r in requests] # back in input order
class Req:
def __init__(self, i):
self.id = i
async def fake_client_run(reqs):
fake_client_run.calls += len(reqs)
await asyncio.sleep(0.01)
return [{"id": r.id, "score": r.id % 10} for r in reqs]
fake_client_run.calls = 0
async def main():
requests = [Req(i) for i in range(20)]
first = await run_with_checkpoint(fake_client_run, requests, done={})
half = {res["id"]: res for res in first if res["id"] % 2 == 0} # pretend the odd half was lost
calls_before = fake_client_run.calls
second = await run_with_checkpoint(fake_client_run, requests, done=half)
print("provider calls on re-run:", fake_client_run.calls - calls_before)
assert second == first
print("results identical and in order")
asyncio.run(main())
The re-run calls the provider only 10 times, for the missing ids. In the real client, done comes from reading the JSONL checkpoint file at start-up, and each result is appended to that file the moment it finishes.
You've finished the core Python path. From here:
- PyTorch engineering — tensors, autograd and training loops, built on NumPy and the performance ideas here.
- High-throughput networking — what's under
await: sockets, connection pools, backpressure and retries at scale.