The incident. A team switches an evaluation job to async and fires all 20,000 requests at once with asyncio.gather. Within seconds, the provider returns thousands of 429 Too Many Requests errors, the retries pile on top, and the organisation's API key is throttled for the rest of the day, taking down the production chatbot that shares it. Async made them fast. Nothing made them polite.
You already know: I/O-bound work releases the GIL, and threads overlap waiting (The GIL); decorators and context managers; profiling showed the time is spent waiting on the network.
1. The event loop: one thread, many waits
The question. How can one thread handle thousands of network calls at once?
Start from nothing. Most of an API call is waiting. asyncio runs an event loop on one thread. An async def function is a coroutine: when it reaches await on something slow (a network response, a sleep), it pauses and hands control back to the loop, which runs another coroutine that's ready. When the response arrives, the paused coroutine resumes. No threads, no locks for most code, and each waiting task costs a few KB instead of a thread's megabytes.
import asyncio, time
async def call_model(prompt, seconds):
await asyncio.sleep(seconds) # stands in for waiting on an HTTP response
return f"{prompt}: done"
async def main():
start = time.perf_counter()
one = await call_model("warm-up", 0.3) # await: wait for this one
print(one, f"({time.perf_counter() - start:.1f}s)")
start = time.perf_counter()
results = await asyncio.gather( # run all three concurrently
call_model("a", 0.5), call_model("b", 0.5), call_model("c", 0.5),
)
print(results, f"({time.perf_counter() - start:.1f}s)")
asyncio.run(main()) # start the event loop
The three 0.5-second calls finish together in about 0.5 seconds, not 1.5. Two rules:
- Calling
call_model(...)withoutawaitonly creates a coroutine; nothing runs until it's awaited or scheduled. - Never block the loop:
time.sleep,requests.getor heavy CPU work insideasync deffreezes every task. Use async libraries (httpx.AsyncClient,asyncio.sleep) or move blocking work to a thread withasyncio.to_thread.
From production
An async FastAPI endpoint calls requests.post(llm_url, ...) (the regular, synchronous requests library). Under load, p99 latency for *all* endpoints jumps to 30 seconds.
Why?
Show the answer
C. requests.post blocks the event loop thread, so no other request can progress while it waits — A blocking call inside async code stops the whole event loop: every other coroutine waits too. Use an async client (httpx.AsyncClient) or offload with asyncio.to_thread.
2. Limit concurrency with a Semaphore
The question. How do you go fast without overwhelming the API?
Start from nothing. A semaphore is a counter of permits. async with sem: takes a permit (waiting if none are free) and returns it on exit, even on errors. With Semaphore(10), at most 10 requests are in flight at any moment, however many tasks you create. This is the incident's fix:
import asyncio, time
async def call_model(i):
await asyncio.sleep(0.2)
return i * i
async def main():
sem = asyncio.Semaphore(10)
in_flight = peak = 0
async def limited(i):
nonlocal in_flight, peak
async with sem: # at most 10 tasks get past this line at once
in_flight += 1
peak = max(peak, in_flight)
try:
return await call_model(i)
finally:
in_flight -= 1
start = time.perf_counter()
results = await asyncio.gather(*(limited(i) for i in range(100)))
print(f"{len(results)} calls in {time.perf_counter() - start:.1f}s, peak in flight: {peak}")
asyncio.run(main())
100 calls of 0.2 s with 10 at a time take about 2 seconds, and never more than 10 run at once. Pick the limit from the provider's documented limits, then measure.
How to choose the number. Little's law links concurrency, throughput and latency: requests in flight ≈ requests per second × seconds per request. If an LLM call takes about 4 seconds and you want 5 completed calls per second, you need about 20 in flight. Set the semaphore there, then check that 5 per second fits under the provider's requests-per-minute limit (300 per minute here). Going higher than the provider allows doesn't make the job faster. It converts throughput into 429 errors.
In production. LLM providers publish limits per API key or organisation, typically requests per minute and tokens per minute, and some count input and output tokens separately. The official Python SDKs include async clients (for example
AsyncAnthropic), so the same semaphore pattern wraps real calls directly. A shared key across services is a classic outage path: one batch job saturates the limit, and the user-facing product starts failing. Give batch jobs their own key or a strictly lower concurrency.
From production
Each embedding call takes ~0.5 s. You need to embed 600,000 documents within 2 hours, and the provider allows 5,000 requests per minute.
Roughly what semaphore limit do you need, and is it allowed?
Show the answer
D. About 42 in flight (83 req/s × 0.5 s), which is about 5,000 requests per minute: right at the provider's limit, so batch several texts per request or ask for a higher limit — 600,000 / 7,200 s ≈ 83 requests per second. Little's law: 83 × 0.5 ≈ 42 in flight. 83 req/s ≈ 5,000 per minute, exactly the cap, with no headroom for retries. Batch several texts per call (embedding APIs accept lists) to cut the request count.
3. Timeouts, errors and retries
The question. What happens when one call hangs, or ten of them fail?
Start from nothing. Production async code needs three more things:
- Timeouts:
asyncio.wait_for(coro, timeout=...)cancels a call that takes too long. A hung connection otherwise holds a semaphore permit forever. - Error isolation:
gather(..., return_exceptions=True)returns exceptions as results instead of failing everything on the first error. - Retries with backoff and jitter for transient errors (429, 503, timeouts), never for errors like 400 Bad Request.
import asyncio, random
class RateLimited(Exception):
pass
async def flaky_call(i, rng):
roll = rng.random()
if roll < 0.2:
raise RateLimited(f"429 on {i}")
if roll < 0.3:
await asyncio.sleep(5) # a hung request
await asyncio.sleep(0.05)
return f"ok {i}"
async def with_retries(i, rng, attempts=4):
for attempt in range(attempts):
try:
return await asyncio.wait_for(flaky_call(i, rng), timeout=0.5)
except (RateLimited, asyncio.TimeoutError):
if attempt == attempts - 1:
raise
await asyncio.sleep(0.05 * 2 ** attempt * random.uniform(0.5, 1.5)) # backoff + jitter
async def main():
rng = random.Random(11)
results = await asyncio.gather(*(with_retries(i, rng) for i in range(30)), return_exceptions=True)
ok = sum(isinstance(r, str) for r in results)
failed = [r for r in results if isinstance(r, Exception)]
print(f"{ok} ok, {len(failed)} failed after retries")
asyncio.run(main())
In this run all 30 calls succeed, some only after a retry or two; each hung call is cut off at 0.5 s instead of blocking for 5. Anything that still fails after four attempts is returned as an exception, which you log and send to a retry queue, while the other results are kept.
In production. Long batch jobs (evaluations, backfills, bulk embedding) also checkpoint: they write each result as it completes, for example as JSONL lines keyed by item ID, and skip finished IDs on restart. A crash at item 4,812 then costs one item, not hours of work. Some providers also offer batch APIs that accept a file of requests and return results asynchronously at a lower price, which is often the right tool when nobody is waiting on the answer.
From production
A batch job uses await asyncio.gather(*tasks) over 5,000 API calls. Call #4,812 raises a ValueError on a malformed response.
What happens by default, and what's the safer pattern?
Show the answer
A. gather raises the first exception to the caller (results of the others are lost to your code); use return_exceptions=True or handle errors per task and record failures — By default, gather propagates the first exception, and your code never receives the other results, even though they completed. Catch errors inside each task (or use return_exceptions=True), so one bad response doesn't throw away hours of work.
4. Rate limits: a token bucket
The question. A semaphore limits how many calls run at once. How do you limit calls per second?
Start from nothing. Providers usually limit requests per minute (and tokens per minute). A token bucket models this: the bucket holds up to capacity tokens and refills at rate tokens per second. Each request takes one token, waiting if the bucket is empty. Bursts up to the capacity are allowed; the long-run average never exceeds the rate.
import asyncio, time
class TokenBucket:
def __init__(self, rate, capacity):
self.rate, self.capacity = rate, capacity
self.tokens = capacity
self.updated = time.monotonic()
self.lock = asyncio.Lock()
async def acquire(self):
async with self.lock: # one coroutine updates the bucket at a time
while True:
now = time.monotonic()
self.tokens = min(self.capacity, self.tokens + (now - self.updated) * self.rate)
self.updated = now
if self.tokens >= 1:
self.tokens -= 1
return
await asyncio.sleep((1 - self.tokens) / self.rate)
async def main():
bucket = TokenBucket(rate=20, capacity=5) # 20 requests/second, bursts of 5
start = time.perf_counter()
stamps = []
async def request(i):
await bucket.acquire()
stamps.append(time.perf_counter() - start)
await asyncio.gather(*(request(i) for i in range(45)))
print(f"45 requests in {stamps[-1]:.2f}s (5 burst + 40 at 20/s = ~2.0s)")
asyncio.run(main())
Five requests go immediately (the burst), and the remaining 40 are spaced at 20 per second, so the run takes about 2 seconds. Combine both limits in real clients: the bucket for requests per second, a semaphore for requests in flight. When several processes share one API key, the bucket has to live in a shared store like Redis.
In production. For CPU-heavy steps inside an async service (parsing PDFs, tokenizing big batches), use
loop.run_in_executorwith a process pool, or a separate worker service. Async overlaps waiting; it doesn't make computation parallel.
From production
Your provider allows 500 requests per minute. Three worker processes each run an in-memory token bucket set to 500/min.
What happens?
Show the answer
A. Together they can send up to 1,500 per minute and get rate-limited; the limit must be shared (e.g. a Redis-backed bucket) or split (about 166/min each) — Each in-memory bucket only knows about its own process. The provider counts all requests from the key. Share the bucket's state, or divide the budget between workers.
Check yourself
- What does
awaitdo to the coroutine that runs it? - Why is
time.sleep(1)insideasync defa serious bug? - What's the difference between a semaphore and a token bucket?
- Why use
asyncio.wait_foraround network calls? - Calls take 2 seconds and you want 10 completed per second. Roughly how many must be in flight?
- Why should a long batch job write each result as it completes?
Answers
- Pauses it until the awaited thing finishes, letting the event loop run other tasks in the meantime.
- It blocks the event loop thread, so every other task freezes for that second. Use
await asyncio.sleep(1). - A semaphore caps how many operations run at once; a token bucket caps how many start per second (with bursts up to its capacity).
- A hung request would otherwise wait forever, holding resources such as a semaphore permit; a timeout turns it into an error you can retry.
- About 20 (Little's law: 10 per second × 2 seconds), provided that rate fits under the provider's limits.
- So a crash or restart can skip finished items instead of redoing (and re-paying for) hours of work.
Exercise
Write async def map_limited(fn, items, concurrency) that runs fn(item) for every item with at most concurrency in flight, returns results in the same order as items, and records failures as exceptions instead of stopping. Test it with 50 items, a concurrency of 8, and a function that fails for multiples of 13.
Solution
import asyncio, time
async def map_limited(fn, items, concurrency):
sem = asyncio.Semaphore(concurrency)
async def run(item):
async with sem:
try:
return await fn(item)
except Exception as err: # record the failure, keep going
return err
return await asyncio.gather(*(run(x) for x in items)) # gather keeps input order
async def score(i):
await asyncio.sleep(0.05)
if i % 13 == 0 and i:
raise ValueError(f"bad item {i}")
return i * 10
async def main():
start = time.perf_counter()
results = await map_limited(score, range(50), concurrency=8)
elapsed = time.perf_counter() - start
errors = [r for r in results if isinstance(r, Exception)]
print(results[:5], "...", f"{len(errors)} errors:", [str(e) for e in errors])
print(f"{elapsed:.2f}s (50 items / 8 at a time x 0.05s = ~0.35s)")
asyncio.run(main())
gather returns results in the order its arguments were given, regardless of which finished first, so ordering comes free. Items 13, 26 and 39 fail and come back as ValueError objects in their positions. The capstone builds this into a full client with rate limits and retries.
This unlocks:
- Interview prep and capstone — an async, rate-limited batch client for LLM APIs.
- High-throughput networking — what happens below
await: sockets, connection pools and backpressure.