VFlash: Fast, Accurate Visual Planning via Flash Primitive Decoding
u_Unrolled
D1: CS & Systems Foundations · Async Python — Thousands of LLM Calls Without Thousands of Threads
Phase 1/Domain 1/Lesson 17

Async Python — Thousands of LLM Calls Without Thousands of Threads

15 min read·Foundational·Free lesson
Async Python — Thousands of LLM Calls Without Thousands of Threads

The incident. A team switches an evaluation job to async and fires all 20,000 requests at once with asyncio.gather. Within seconds, the provider returns thousands of 429 Too Many Requests errors, the retries pile on top, and the organisation's API key is throttled for the rest of the day, taking down the production chatbot that shares it. Async made them fast. Nothing made them polite.

You already know: I/O-bound work releases the GIL, and threads overlap waiting (The GIL); decorators and context managers; profiling showed the time is spent waiting on the network.


1. The event loop: one thread, many waits

The question. How can one thread handle thousands of network calls at once?

Start from nothing. Most of an API call is waiting. asyncio runs an event loop on one thread. An async def function is a coroutine: when it reaches await on something slow (a network response, a sleep), it pauses and hands control back to the loop, which runs another coroutine that's ready. When the response arrives, the paused coroutine resumes. No threads, no locks for most code, and each waiting task costs a few KB instead of a thread's megabytes.

python
import asyncio, time

async def call_model(prompt, seconds):
    await asyncio.sleep(seconds)           # stands in for waiting on an HTTP response
    return f"{prompt}: done"

async def main():
    start = time.perf_counter()
    one = await call_model("warm-up", 0.3)              # await: wait for this one
    print(one, f"({time.perf_counter() - start:.1f}s)")

    start = time.perf_counter()
    results = await asyncio.gather(                     # run all three concurrently
        call_model("a", 0.5), call_model("b", 0.5), call_model("c", 0.5),
    )
    print(results, f"({time.perf_counter() - start:.1f}s)")

asyncio.run(main())                                     # start the event loop

The three 0.5-second calls finish together in about 0.5 seconds, not 1.5. Two rules:

  • Calling call_model(...) without await only creates a coroutine; nothing runs until it's awaited or scheduled.
  • Never block the loop: time.sleep, requests.get or heavy CPU work inside async def freezes every task. Use async libraries (httpx.AsyncClient, asyncio.sleep) or move blocking work to a thread with asyncio.to_thread.
an event loop timeline: tasks A, B and C each start a request, pause at await, and the loop switches between them, so the waits overlap on a single thread.
Diagram— an event loop timeline: tasks A, B and C each start a request, pause at await, and the loop switches between them, so the waits overlap on a single thread.

From production

An async FastAPI endpoint calls requests.post(llm_url, ...) (the regular, synchronous requests library). Under load, p99 latency for *all* endpoints jumps to 30 seconds.

Why?

  1. FastAPI is slow
  2. async functions can't make HTTP calls
  3. requests.post blocks the event loop thread, so no other request can progress while it waits
  4. The LLM provider is slow only for this endpoint
Show the answer

C. requests.post blocks the event loop thread, so no other request can progress while it waits — A blocking call inside async code stops the whole event loop: every other coroutine waits too. Use an async client (httpx.AsyncClient) or offload with asyncio.to_thread.

2. Limit concurrency with a Semaphore

The question. How do you go fast without overwhelming the API?

Start from nothing. A semaphore is a counter of permits. async with sem: takes a permit (waiting if none are free) and returns it on exit, even on errors. With Semaphore(10), at most 10 requests are in flight at any moment, however many tasks you create. This is the incident's fix:

python
import asyncio, time

async def call_model(i):
    await asyncio.sleep(0.2)
    return i * i

async def main():
    sem = asyncio.Semaphore(10)
    in_flight = peak = 0

    async def limited(i):
        nonlocal in_flight, peak
        async with sem:                      # at most 10 tasks get past this line at once
            in_flight += 1
            peak = max(peak, in_flight)
            try:
                return await call_model(i)
            finally:
                in_flight -= 1

    start = time.perf_counter()
    results = await asyncio.gather(*(limited(i) for i in range(100)))
    print(f"{len(results)} calls in {time.perf_counter() - start:.1f}s, peak in flight: {peak}")

asyncio.run(main())

100 calls of 0.2 s with 10 at a time take about 2 seconds, and never more than 10 run at once. Pick the limit from the provider's documented limits, then measure.

top, gather over 100 unlimited tasks sends all 100 requests in the same instant, leading to 429s and throttling; bottom, with Semaphore(10) the calls run in ten waves of ten (1–10, 11–20, … 91–100), finishing in about 2.0 seconds with a peak of 10 in flight.
Diagram— top, gather over 100 unlimited tasks sends all 100 requests in the same instant, leading to 429s and throttling; bottom, with Semaphore(10) the calls run in ten waves of ten (1–10, 11–20, … 91–100), finishing in about 2.0 seconds with a peak of 10 in flight.

How to choose the number. Little's law links concurrency, throughput and latency: requests in flight ≈ requests per second × seconds per request. If an LLM call takes about 4 seconds and you want 5 completed calls per second, you need about 20 in flight. Set the semaphore there, then check that 5 per second fits under the provider's requests-per-minute limit (300 per minute here). Going higher than the provider allows doesn't make the job faster. It converts throughput into 429 errors.

In production. LLM providers publish limits per API key or organisation, typically requests per minute and tokens per minute, and some count input and output tokens separately. The official Python SDKs include async clients (for example AsyncAnthropic), so the same semaphore pattern wraps real calls directly. A shared key across services is a classic outage path: one batch job saturates the limit, and the user-facing product starts failing. Give batch jobs their own key or a strictly lower concurrency.

From production

Each embedding call takes ~0.5 s. You need to embed 600,000 documents within 2 hours, and the provider allows 5,000 requests per minute.

Roughly what semaphore limit do you need, and is it allowed?

  1. 5 in flight; well under the limit
  2. 600,000 in flight, since async has no limit
  3. 1 in flight; parallelism is never allowed
  4. About 42 in flight (83 req/s × 0.5 s), which is about 5,000 requests per minute: right at the provider's limit, so batch several texts per request or ask for a higher limit
Show the answer

D. About 42 in flight (83 req/s × 0.5 s), which is about 5,000 requests per minute: right at the provider's limit, so batch several texts per request or ask for a higher limit — 600,000 / 7,200 s ≈ 83 requests per second. Little's law: 83 × 0.5 ≈ 42 in flight. 83 req/s ≈ 5,000 per minute, exactly the cap, with no headroom for retries. Batch several texts per call (embedding APIs accept lists) to cut the request count.

3. Timeouts, errors and retries

The question. What happens when one call hangs, or ten of them fail?

Start from nothing. Production async code needs three more things:

  • Timeouts: asyncio.wait_for(coro, timeout=...) cancels a call that takes too long. A hung connection otherwise holds a semaphore permit forever.
  • Error isolation: gather(..., return_exceptions=True) returns exceptions as results instead of failing everything on the first error.
  • Retries with backoff and jitter for transient errors (429, 503, timeouts), never for errors like 400 Bad Request.
python
import asyncio, random

class RateLimited(Exception):
    pass

async def flaky_call(i, rng):
    roll = rng.random()
    if roll < 0.2:
        raise RateLimited(f"429 on {i}")
    if roll < 0.3:
        await asyncio.sleep(5)              # a hung request
    await asyncio.sleep(0.05)
    return f"ok {i}"

async def with_retries(i, rng, attempts=4):
    for attempt in range(attempts):
        try:
            return await asyncio.wait_for(flaky_call(i, rng), timeout=0.5)
        except (RateLimited, asyncio.TimeoutError):
            if attempt == attempts - 1:
                raise
            await asyncio.sleep(0.05 * 2 ** attempt * random.uniform(0.5, 1.5))   # backoff + jitter

async def main():
    rng = random.Random(11)
    results = await asyncio.gather(*(with_retries(i, rng) for i in range(30)), return_exceptions=True)
    ok = sum(isinstance(r, str) for r in results)
    failed = [r for r in results if isinstance(r, Exception)]
    print(f"{ok} ok, {len(failed)} failed after retries")

asyncio.run(main())

In this run all 30 calls succeed, some only after a retry or two; each hung call is cut off at 0.5 s instead of blocking for 5. Anything that still fails after four attempts is returned as an exception, which you log and send to a retry queue, while the other results are kept.

each call runs under wait_for(call, 0.5 s); "ok" keeps the result, "429/timeout" goes to a backoff-with-jitter sleep and loops back while fewer than 4 attempts have been made, "400 Bad Request" is never retried; gather(*tasks, return_exceptions=True) returns 30 results, with exception objects for final failures.
Diagram— each call runs under wait_for(call, 0.5 s); "ok" keeps the result, "429/timeout" goes to a backoff-with-jitter sleep and loops back while fewer than 4 attempts have been made, "400 Bad Request" is never retried; gather(*tasks, return_exceptions=True) returns 30 results, with exception objects for final failures.

In production. Long batch jobs (evaluations, backfills, bulk embedding) also checkpoint: they write each result as it completes, for example as JSONL lines keyed by item ID, and skip finished IDs on restart. A crash at item 4,812 then costs one item, not hours of work. Some providers also offer batch APIs that accept a file of requests and return results asynchronously at a lower price, which is often the right tool when nobody is waiting on the answer.

From production

A batch job uses await asyncio.gather(*tasks) over 5,000 API calls. Call #4,812 raises a ValueError on a malformed response.

What happens by default, and what's the safer pattern?

  1. gather raises the first exception to the caller (results of the others are lost to your code); use return_exceptions=True or handle errors per task and record failures
  2. The other 4,999 tasks are rolled back
  3. Python retries the failed task automatically
  4. The ValueError is ignored
Show the answer

A. gather raises the first exception to the caller (results of the others are lost to your code); use return_exceptions=True or handle errors per task and record failures — By default, gather propagates the first exception, and your code never receives the other results, even though they completed. Catch errors inside each task (or use return_exceptions=True), so one bad response doesn't throw away hours of work.

4. Rate limits: a token bucket

The question. A semaphore limits how many calls run at once. How do you limit calls per second?

Start from nothing. Providers usually limit requests per minute (and tokens per minute). A token bucket models this: the bucket holds up to capacity tokens and refills at rate tokens per second. Each request takes one token, waiting if the bucket is empty. Bursts up to the capacity are allowed; the long-run average never exceeds the rate.

python
import asyncio, time

class TokenBucket:
    def __init__(self, rate, capacity):
        self.rate, self.capacity = rate, capacity
        self.tokens = capacity
        self.updated = time.monotonic()
        self.lock = asyncio.Lock()

    async def acquire(self):
        async with self.lock:                 # one coroutine updates the bucket at a time
            while True:
                now = time.monotonic()
                self.tokens = min(self.capacity, self.tokens + (now - self.updated) * self.rate)
                self.updated = now
                if self.tokens >= 1:
                    self.tokens -= 1
                    return
                await asyncio.sleep((1 - self.tokens) / self.rate)

async def main():
    bucket = TokenBucket(rate=20, capacity=5)          # 20 requests/second, bursts of 5
    start = time.perf_counter()
    stamps = []

    async def request(i):
        await bucket.acquire()
        stamps.append(time.perf_counter() - start)

    await asyncio.gather(*(request(i) for i in range(45)))
    print(f"45 requests in {stamps[-1]:.2f}s (5 burst + 40 at 20/s = ~2.0s)")

asyncio.run(main())

Five requests go immediately (the burst), and the remaining 40 are spaced at 20 per second, so the run takes about 2 seconds. Combine both limits in real clients: the bucket for requests per second, a semaphore for requests in flight. When several processes share one API key, the bucket has to live in a shared store like Redis.

a bucket with capacity 5 refilling at 20 tokens per second; of 45 requests arriving together, 5 go at once (the burst), then one every 50 ms, with the 45th at about 2.0 s; below, three processes each running a 500/min in-memory bucket can send 1,500/min on one key.
Diagram— a bucket with capacity 5 refilling at 20 tokens per second; of 45 requests arriving together, 5 go at once (the burst), then one every 50 ms, with the 45th at about 2.0 s; below, three processes each running a 500/min in-memory bucket can send 1,500/min on one key.

In production. For CPU-heavy steps inside an async service (parsing PDFs, tokenizing big batches), use loop.run_in_executor with a process pool, or a separate worker service. Async overlaps waiting; it doesn't make computation parallel.

From production

Your provider allows 500 requests per minute. Three worker processes each run an in-memory token bucket set to 500/min.

What happens?

  1. Together they can send up to 1,500 per minute and get rate-limited; the limit must be shared (e.g. a Redis-backed bucket) or split (about 166/min each)
  2. They stay under the limit
  3. Only the first process sends requests
  4. Python merges the buckets automatically
Show the answer

A. Together they can send up to 1,500 per minute and get rate-limited; the limit must be shared (e.g. a Redis-backed bucket) or split (about 166/min each) — Each in-memory bucket only knows about its own process. The provider counts all requests from the key. Share the bucket's state, or divide the budget between workers.


Check yourself

  1. What does await do to the coroutine that runs it?
  2. Why is time.sleep(1) inside async def a serious bug?
  3. What's the difference between a semaphore and a token bucket?
  4. Why use asyncio.wait_for around network calls?
  5. Calls take 2 seconds and you want 10 completed per second. Roughly how many must be in flight?
  6. Why should a long batch job write each result as it completes?
Answers
  1. Pauses it until the awaited thing finishes, letting the event loop run other tasks in the meantime.
  2. It blocks the event loop thread, so every other task freezes for that second. Use await asyncio.sleep(1).
  3. A semaphore caps how many operations run at once; a token bucket caps how many start per second (with bursts up to its capacity).
  4. A hung request would otherwise wait forever, holding resources such as a semaphore permit; a timeout turns it into an error you can retry.
  5. About 20 (Little's law: 10 per second × 2 seconds), provided that rate fits under the provider's limits.
  6. So a crash or restart can skip finished items instead of redoing (and re-paying for) hours of work.

Exercise

Write async def map_limited(fn, items, concurrency) that runs fn(item) for every item with at most concurrency in flight, returns results in the same order as items, and records failures as exceptions instead of stopping. Test it with 50 items, a concurrency of 8, and a function that fails for multiples of 13.

Solution
python
import asyncio, time

async def map_limited(fn, items, concurrency):
    sem = asyncio.Semaphore(concurrency)
    async def run(item):
        async with sem:
            try:
                return await fn(item)
            except Exception as err:            # record the failure, keep going
                return err
    return await asyncio.gather(*(run(x) for x in items))   # gather keeps input order

async def score(i):
    await asyncio.sleep(0.05)
    if i % 13 == 0 and i:
        raise ValueError(f"bad item {i}")
    return i * 10

async def main():
    start = time.perf_counter()
    results = await map_limited(score, range(50), concurrency=8)
    elapsed = time.perf_counter() - start
    errors = [r for r in results if isinstance(r, Exception)]
    print(results[:5], "...", f"{len(errors)} errors:", [str(e) for e in errors])
    print(f"{elapsed:.2f}s (50 items / 8 at a time x 0.05s = ~0.35s)")

asyncio.run(main())

gather returns results in the order its arguments were given, regardless of which finished first, so ordering comes free. Items 13, 26 and 39 fail and come back as ValueError objects in their positions. The capstone builds this into a full client with rate limits and retries.


This unlocks:

  • Interview prep and capstone — an async, rate-limited batch client for LLM APIs.
  • High-throughput networking — what happens below await: sockets, connection pools and backpressure.
Async Python — Thousands of LLM Calls Without Thousands of Threads · Unrolled