The incident. An inference service opens a GPU memory pool and a database connection per request, and closes them at the end of the function. When a request raises an exception halfway, the cleanup lines never run. After two days, the database refuses connections ("too many clients") and the GPU reports out of memory. Nothing was slow; resources simply leaked on every error path.
You already know: functions are objects and scope follows LEGB (Functions); generators pause at yield (Iterators and generators).
1. Closures: functions that remember
The question. How can a function keep state between calls without a global variable?
Start from nothing. When an inner function uses a name from its enclosing function, Python keeps that variable alive even after the outer function returns. The inner function plus the variables it captured is a closure. Use nonlocal to reassign a captured variable.
def make_counter(label):
count = 0
def increment():
nonlocal count # reassign the enclosing variable, not a new local
count += 1
return f"{label}: {count}"
return increment
api_calls = make_counter("api")
db_calls = make_counter("db")
print(api_calls(), api_calls(), db_calls())
print(api_calls.__closure__[0].cell_contents) # where the captured value lives
This prints api: 1 api: 2 db: 1, then 2. Each counter has its own count, with no globals.
A real-world example: a token budget for an agent. Agents call a model in a loop, and a bug or an adversarial input can make them loop for a long time. A closure gives each agent run its own spending guard, with no global state and nothing else able to tamper with it:
class BudgetExceeded(Exception):
pass
def make_budget(limit_tokens):
spent = 0
def charge(tokens):
nonlocal spent
if spent + tokens > limit_tokens:
raise BudgetExceeded(f"would spend {spent + tokens:,} of {limit_tokens:,} tokens")
spent += tokens
return limit_tokens - spent # tokens left
return charge
agent_budget = make_budget(10_000)
for step_tokens in [3_000, 4_500, 2_000, 1_800]:
try:
print("left:", agent_budget(step_tokens))
except BudgetExceeded as e:
print("stopped:", e)
break
The first three steps leave 7000, 2500 and 500 tokens. The fourth would bring the total to 11,300, so the guard stops the run before spending: stopped: would spend 11,300 of 10,000 tokens. Checking before charging matters: the check runs before the expensive call, not after the money is gone.
In production. Per-run budgets (tokens, dollars, tool calls, wall-clock time) are standard guardrails in agent frameworks and LLM gateways, because one runaway loop can cost more than a day of normal traffic. A closure is the simplest form. In a real service the counter is usually shared across processes (Redis or a database row) so a budget holds across many workers. The idea stays the same: state captured once, checked on every call.
From production
Two concurrent agent runs are each created with charge = make_budget(10_000).
How do their budgets interact?
Show the answer
C. Each call to make_budget creates a new spent variable, so each run has its own independent 10,000-token budget — Every call to the factory creates fresh local variables, and each returned charge() closes over its own cell. That's exactly why closures give private, per-instance state without globals. (Within one run, concurrent threads would still need a lock around the read-modify-write.)
2. Decorators: wrapping a function
The question. How do you add timing or retries to many functions without editing each one?
Start from nothing. A decorator is a function that takes a function and returns a new function, usually a closure that calls the original with something extra around it. @timer above a def is shorthand for fn = timer(fn). Always apply functools.wraps, so the wrapper keeps the original's name and docstring (logs, tracebacks and frameworks depend on them).
import functools, time, random
def timer(fn):
@functools.wraps(fn)
def wrapper(*args, **kwargs):
start = time.perf_counter()
try:
return fn(*args, **kwargs)
finally:
print(f"{fn.__name__} took {(time.perf_counter() - start) * 1000:.0f} ms")
return wrapper
def retry(times=3, exceptions=(ConnectionError,)): # a decorator factory: takes settings
def decorate(fn):
@functools.wraps(fn)
def wrapper(*args, **kwargs):
for attempt in range(1, times + 1):
try:
return fn(*args, **kwargs)
except exceptions as err:
print(f" attempt {attempt} failed: {err}")
if attempt == times:
raise
return wrapper
return decorate
random.seed(7)
@timer
@retry(times=4)
def call_model(prompt):
"""Pretend LLM call that fails sometimes."""
time.sleep(0.05)
if random.random() < 0.5:
raise ConnectionError("upstream reset")
return f"answer to {prompt!r}"
print(call_model("what is a closure?"))
print(call_model.__name__, "|", call_model.__doc__)
- Decorators stack bottom-up:
retrywraps the function first, thentimerwraps the retrying version, so the time includes every retry. retry(times=4)is a decorator factory: it takes settings and returns the actual decorator.- Thanks to
wraps, the name and docstring survive:call_model | Pretend LLM call that fails sometimes.
From production
A team's @retry decorator retries on any Exception, 5 times, with no delay. It wraps a function that charges a customer's card.
What's the biggest danger?
Show the answer
C. A timeout after the charge succeeded is retried, charging the customer up to five times — Retrying is only safe for idempotent operations (doing it twice has the same effect as once). Retry only specific transient errors, add backoff, and for payments use an idempotency key so the provider ignores duplicates.
3. Caching with lru_cache
The question. What if the same expensive call happens again with the same arguments?
Start from nothing. functools.lru_cache is a ready-made decorator that remembers results by argument. LRU (least recently used) means that when it's full, the entry unused for the longest gets evicted. Arguments must be hashable.
from functools import lru_cache
import time
@lru_cache(maxsize=1024)
def embed(text):
time.sleep(0.1) # stands in for a slow embedding call
return hash(text) % 1000
start = time.perf_counter()
for q in ["refund policy", "reset password", "refund policy", "refund policy"]:
embed(q)
print(f"{time.perf_counter() - start:.2f}s", embed.cache_info())
Only 2 of the 4 calls do real work: the output shows about 0.20s and CacheInfo(hits=2, misses=2, maxsize=1024, currsize=2). Always set maxsize: an unbounded cache in a long-running server is a memory leak with a nice name.
Two caching traps that show up in code review. First, @lru_cache on a method caches on (self, args), so the cache holds a reference to every instance it has seen. Instances that should be garbage-collected stay alive. Second, a cache has no idea when the underlying data changes.
import gc, weakref
from functools import lru_cache
class Retriever:
def __init__(self, name):
self.name = name
@lru_cache(maxsize=128) # caches on (self, query): the cache holds self
def search(self, query):
return f"{self.name}:{query}"
r = Retriever("index-v1")
r.search("refund policy")
alive = weakref.ref(r)
del r
gc.collect()
print("instance still alive after del:", alive() is not None)
print(Retriever.search.cache_info())
After del r and a full garbage collection, the instance is still alive (True). The class-level cache holds it in its one entry (currsize=1). If each instance holds a loaded index or model, that's a real memory leak. The bugbear linter flags this pattern (rule B019). The usual fixes are a cache stored on the instance, or caching a module-level function of plain arguments.
In production.
lru_cacheis right for pure functions of hashable inputs: parsing a config, tokenizing a fixed prompt template, computing a schema. For data that changes, such as user profiles, documents being edited, or prices, you need expiry (a TTL cache likecachetools.TTLCache) or explicit invalidation. Across many processes you need a shared cache such as Redis. Embedding caches are a common real case: key them on a hash of the text and the embedding model's name, so that switching models never serves vectors from the old one.
From production
A RAG service caches embed(text) with @lru_cache(maxsize=None). After upgrading to a new embedding model, retrieval quality collapses for common queries but not for rare ones.
What's the most likely cause?
Show the answer
C. Common queries hit the old cache and return vectors from the previous model, which live in a different vector space; include the model name in the cache key (and bound the cache) — The cache key is just the text, so popular texts keep returning old-model vectors that don't match the new index. Key on (model_name, text_hash), set a maxsize, and clear caches on model changes.
4. Context managers: cleanup that always happens
The question. How do you guarantee cleanup, even when an exception is raised?
Start from nothing. A context manager runs setup on entering a with block and cleanup on leaving it, however it's left: normally, by return, or by an exception. Files, locks and DB connections all support it. The easiest way to write one is contextlib.contextmanager: everything before yield is setup, and the finally after it is the cleanup.
from contextlib import contextmanager
POOL = {"open": 0}
@contextmanager
def connection(name):
POOL["open"] += 1
print(f"open {name} (open: {POOL['open']})")
try:
yield f"<conn {name}>" # the value bound by `as`
finally:
POOL["open"] -= 1 # runs even if the block raises
print(f"close {name} (open: {POOL['open']})")
with connection("db") as conn:
print("using", conn)
try:
with connection("db") as conn:
raise TimeoutError("query took too long")
except TimeoutError as err:
print("handled:", err)
print("connections still open:", POOL["open"])
Both blocks close their connection; the second closes it before the exception reaches the except. The final count is 0: this is the incident's fix. The class-based form defines __enter__ and __exit__, which the next lesson's classes make easy.
Context managers you'll write in real services. Two patterns cover most needs. A timing block records how long each stage took, including stages that failed. Failures are often the slow ones you most need to see. An ExitStack manages a number of resources you only know at run time, such as one output file per shard:
import os, tempfile, time
from contextlib import contextmanager, ExitStack
@contextmanager
def timed(stage, log):
start = time.perf_counter()
try:
yield
finally:
log[stage] = round((time.perf_counter() - start) * 1000) # recorded even on failure
log = {}
with timed("retrieve", log):
time.sleep(0.03)
try:
with timed("generate", log):
time.sleep(0.02)
raise TimeoutError("model timed out")
except TimeoutError:
pass
print(sorted(log)) # both stages were timed
folder = tempfile.mkdtemp()
paths = [os.path.join(folder, f"shard-{i}.jsonl") for i in range(3)]
with ExitStack() as stack: # a variable number of resources, all closed together
files = [stack.enter_context(open(p, "w", encoding="utf-8")) for p in paths]
for i, f in enumerate(files):
f.write(f'{{"shard": {i}}}\n')
print("all closed:", all(f.closed for f in files))
Both stages appear in the log, ['generate', 'retrieve'], even though generate raised. The finally block recorded its duration on the way out. All three shard files are closed after the with (True). If opening the third file had failed, ExitStack would still have closed the first two.
In production. You'll use context managers from libraries constantly.
torch.no_grad()andtorch.autocast(...)switch PyTorch modes for a block. Database sessions commit or roll back on exit. Tracing libraries such as OpenTelemetry open a span withwith tracer.start_as_current_span("retrieve"):, which is the timing block above, made distributed. Async code hasasync withfor connections and HTTP clients (Async Python for LLM workloads).
From production
A GPU service does lock.acquire(), runs inference, then lock.release(). Occasionally inference raises CUDA OOM, and afterwards every request hangs forever.
Why, and what's the fix?
Show the answer
A. The exception skips release(), so the lock is held forever; use with lock: so it's released on any exit — Code after the failing line never runs, so the lock stays acquired and every later request waits on it. with lock: releases it on normal exit and on exceptions.
Check yourself
- What is a closure?
- What does
@timerabove a function definition do? - Why use
functools.wrapsin every decorator? - When does the cleanup in a context manager run?
- Why is
@lru_cacheon an instance method a memory-leak risk? - Why should an embedding cache key include the model name?
Answers
- An inner function together with variables it captured from an enclosing function, which stay alive after the outer function returns.
- It replaces the function with
timer(function), usually a wrapper that runs extra code around the original. - It copies the original's name, docstring and other metadata onto the wrapper, so logs, debugging and frameworks still see the real function.
- Whenever the
withblock is left: normally, through return or break, or because of an exception. - The cache key includes
self, so the class-level cache keeps a reference to every instance it has seen, which then can't be garbage-collected. - Different models produce vectors in different spaces. Keying on text alone would serve old-model vectors after an upgrade.
Exercise
Write a decorator rate_limited(calls, per_seconds) that blocks (sleeps) so the wrapped function is called at most calls times in any per_seconds window. Test that 5 calls with rate_limited(2, 0.5) take at least 1 second.
Solution
import functools, time
from collections import deque
def rate_limited(calls, per_seconds):
def decorate(fn):
recent = deque() # start times of recent calls
@functools.wraps(fn)
def wrapper(*args, **kwargs):
now = time.monotonic()
while recent and now - recent[0] >= per_seconds:
recent.popleft() # forget calls outside the window
if len(recent) >= calls:
time.sleep(per_seconds - (now - recent[0]))
recent.popleft()
recent.append(time.monotonic())
return fn(*args, **kwargs)
return wrapper
return decorate
@rate_limited(2, 0.5)
def ping(i):
return i
start = time.monotonic()
results = [ping(i) for i in range(5)]
elapsed = time.monotonic() - start
print(results, f"{elapsed:.2f}s")
assert elapsed >= 1.0
Calls 1–2 run at once, 3–4 wait for the first window to pass (~0.5 s), and call 5 waits again (~1.0 s). time.monotonic() is used because the wall clock can jump (NTP adjustments), while a monotonic clock only moves forward. This decorator isn't thread-safe; the async lesson builds a shared limiter.
This unlocks:
- Classes, dataclasses and type hints — objects with their own behaviour, including
__enter__/__exit__. - Errors, logging and testing — catching the right exceptions and proving your decorators work.
- Async Python —
async withand async decorators for concurrent API calls.