VFlash: Fast, Accurate Visual Planning via Flash Primitive Decoding
u_Unrolled
D1: CS & Systems Foundations · Errors, Logging, Tests and Packaging — Code You Can Trust
Phase 1/Domain 1/Lesson 14

Errors, Logging, Tests and Packaging — Code You Can Trust

15 min read·Foundational·Free lesson
Errors, Logging, Tests and Packaging — Code You Can Trust

The incident. A nightly embedding job "succeeds" for 11 days, but the search index is quietly going stale. The cause: except Exception: pass around the upload call. The vector database had changed its auth, every upload raised an error, and the code swallowed every one. No alert, no log line, no failed run.

You already know: functions, classes and dataclasses (Classes, dataclasses and type hints); context managers (Decorators and context managers).


1. Exceptions: fail loudly, catch narrowly

The question. When should code catch an exception, and what should it do?

Start from nothing. An exception is Python's way to say "this can't continue normally". It travels up the call stack until some try/except catches it, or the program stops with a traceback. Good habits:

  • Catch only what you can handle, and catch specific types (KeyError, TimeoutError), never a bare except:.
  • else runs when no exception happened; finally always runs.
  • Define your own exception types so callers can tell "bad input" from "the provider is down".
  • When converting errors, use raise NewError(...) from err so the original cause stays in the traceback.
python
class ProviderError(Exception):
    """The upstream model provider failed; worth retrying later."""

class BadRequest(Exception):
    """The caller sent something invalid; retrying won't help."""

def parse_tokens(raw):
    try:
        value = int(raw)
    except ValueError as err:
        raise BadRequest(f"max_tokens must be an integer, got {raw!r}") from err
    if value <= 0:
        raise BadRequest("max_tokens must be positive")
    return value

for raw in ["256", "lots", "-5"]:
    try:
        tokens = parse_tokens(raw)
    except BadRequest as err:
        print("400:", err, "| cause:", repr(err.__cause__))
    else:
        print("ok:", tokens)

The output: ok: 256, then 400: max_tokens must be an integer, got 'lots' | cause: ValueError(...), then 400: max_tokens must be positive | cause: None. An API layer can map BadRequest to HTTP 400 and ProviderError to 503 with a retry.

an AppError base with BadRequest (400, fix the request), RateLimited (429, retry later) and ProviderDown (503, retry later); a KeyError that isn't an AppError maps to 500 and an alert, while except Exception: pass would turn all four into silence.
Diagram— an AppError base with BadRequest (400, fix the request), RateLimited (429, retry later) and ProviderDown (503, retry later); a KeyError that isn't an AppError maps to 500 and an alert, while except Exception: pass would turn all four into silence.

A real-world pattern: one place that turns exceptions into responses. Services usually define a small hierarchy with a common base, and put the decision "what do we tell the caller, and is it worth retrying?" on the exception types themselves. Anything that isn't one of your deliberate errors is, by definition, a bug:

python
class AppError(Exception):
    """Base class: everything our service raises on purpose."""
    status, retryable = 500, False

class BadRequest(AppError):
    status, retryable = 400, False

class RateLimited(AppError):
    status, retryable = 429, True

class ProviderDown(AppError):
    status, retryable = 503, True

def to_response(err):
    if isinstance(err, AppError):
        return err.status, ("retry later" if err.retryable else "fix the request"), str(err)
    return 500, "bug: page someone", type(err).__name__     # anything unexpected is a bug

for err in [BadRequest("max_tokens must be positive"), RateLimited("60 rpm exceeded"),
            ProviderDown("upstream 502"), KeyError("api_key")]:
    print(to_response(err))

Each deliberate error maps to its status and advice: (400, 'fix the request', …), (429, 'retry later', …), (503, 'retry later', …). The KeyError falls through to (500, 'bug: page someone', 'KeyError'). Note that the response carries only the exception's type there, not its message, so internal details (here, the name of a secret) don't leak to the caller.

In production. Web frameworks have a hook for exactly this, such as FastAPI's exception handlers and Flask's errorhandler. LLM SDKs ship typed error hierarchies of their own. The Anthropic Python SDK, for example, raises distinct classes for rate limits, connection failures and other API status errors, so your retry logic can catch RateLimitError without also retrying a malformed request. Retry only what's marked retryable, and alert on everything that reaches the "bug" branch.

From production

A code review finds try: upload(batch) except Exception: pass in a nightly job.

What's the best replacement?

  1. Catch the specific transient errors you expect, retry them with backoff, and let anything else fail the run loudly (with a log and an alert)
  2. Leave it; it keeps the job from crashing
  3. Replace pass with print('error')
  4. Wrap the whole job in another try/except
Show the answer

A. Catch the specific transient errors you expect, retry them with backoff, and let anything else fail the run loudly (with a log and an alert) — Swallowing every error turns real failures into silent data loss. Handle the errors you understand; everything else should fail the run, so monitoring notices.

2. Logging instead of print

The question. Why do production systems use logging instead of print?

Start from nothing. The logging module adds levels (DEBUG, INFO, WARNING, ERROR), timestamps and logger names, and lets each environment decide what to show and where to send it, without code changes. logger.exception(...) inside an except records the full traceback. Pass values as arguments (or structured fields), so log systems can search them.

python
import logging, sys

logging.basicConfig(
    stream=sys.stdout, level=logging.INFO,
    format="%(levelname)s %(name)s: %(message)s",
)
log = logging.getLogger("embedder")

log.debug("not shown: level is INFO")
log.info("batch %d uploaded (%d vectors)", 7, 512)
try:
    {}["api_key"]
except KeyError:
    log.exception("upload failed for batch %d", 8)

This prints INFO embedder: batch 7 uploaded (512 vectors), then ERROR embedder: upload failed for batch 8 followed by the full traceback. In production you'd usually emit JSON lines with fields like request_id and user_id, so one request can be traced across services.

Watch out. Never log secrets, API keys or full user prompts containing personal data. Logs are copied to many systems and kept for a long time.

three versions of the same event: print("retrieved 8 chunks") (text only), log.info("retrieved %d chunks", 8) (level, logger, time, filterable), and a JSON log line with request_id and latency_ms fields (searchable and aggregatable); below, what never to log.
Diagram— three versions of the same event: print("retrieved 8 chunks") (text only), log.info("retrieved %d chunks", 8) (level, logger, time, filterable), and a JSON log line with request_id and latency_ms fields (searchable and aggregatable); below, what never to log.

What production logs look like: one JSON object per line. Log platforms such as Datadog, Elasticsearch, Loki and CloudWatch can index fields, but only if you emit fields rather than prose. A small formatter turns every log call into JSON and carries context like the request ID:

python
import json, logging, sys

class JsonFormatter(logging.Formatter):
    def format(self, record):
        entry = {"level": record.levelname, "logger": record.name, "msg": record.getMessage()}
        entry.update(getattr(record, "fields", {}))          # structured context passed via extra=
        return json.dumps(entry)

handler = logging.StreamHandler(sys.stdout)
handler.setFormatter(JsonFormatter())
log = logging.getLogger("rag")
log.addHandler(handler)
log.setLevel(logging.INFO)
log.propagate = False

log.info("retrieved %d chunks", 8, extra={"fields": {"request_id": "req-41", "latency_ms": 132}})
log.warning("context truncated", extra={"fields": {"request_id": "req-41", "dropped_tokens": 950}})

Each call prints one line of JSON, for example {"level": "INFO", "logger": "rag", "msg": "retrieved 8 chunks", "request_id": "req-41", "latency_ms": 132}. Now "show every request where dropped_tokens > 500" or "p95 of latency_ms by hour" is a query, not a grep. Both lines share request_id, so you can follow one user's request through retrieval, generation and any retries.

In production. Libraries such as structlog and python-json-logger do this for you, and OpenTelemetry adds trace and span IDs automatically, so logs line up with distributed traces. For LLM features specifically, teams log prompt metadata (model, token counts, latency, cache hits, which retrieved document IDs were used) rather than the full prompt text, and keep the full text, if at all, in an access-controlled store with a retention limit.

From production

During an incident you need 'all requests from the last hour where the model call took over 5 seconds, grouped by model'. The service logs lines like print(f'call done {model} took {secs}s').

What would have made this a one-minute query?

  1. Printing more lines
  2. Logging at DEBUG level
  3. Structured logs: one JSON object per line with fields like model, latency_ms and request_id, sent to a log platform that indexes them
  4. Writing logs to a local file instead of stdout
Show the answer

C. Structured logs: one JSON object per line with fields like model, latency_ms and request_id, sent to a log platform that indexes them — Free-text logs need fragile regexes to parse. Structured fields can be filtered, grouped and graphed directly, and a request_id lets you follow one request across services.

3. Tests with pytest

The question. How do you know a change didn't break something that used to work?

Start from nothing. A test is code that runs your code and checks the result with assert. pytest finds files named test_*.py, runs every test_* function, and reports failures with clear diffs. Three features cover most needs: plain assert, pytest.raises for expected errors, and parametrize for many cases in one test.

python
# test_tokens.py — run with: pytest -q
import pytest
from myapp.tokens import parse_tokens, BadRequest
from myapp.client import call_with_retry

@pytest.mark.parametrize("raw, expected", [("1", 1), ("256", 256), (" 42 ", 42)])
def test_parses_valid_values(raw, expected):
    assert parse_tokens(raw) == expected

@pytest.mark.parametrize("raw", ["lots", "", "-5", "0"])
def test_rejects_invalid_values(raw):
    with pytest.raises(BadRequest):
        parse_tokens(raw)

def test_retry_gives_up(monkeypatch):
    calls = []
    def always_fails(prompt):
        calls.append(prompt)
        raise ConnectionError("down")
    monkeypatch.setattr("myapp.client.send", always_fails)   # replace the network call
    with pytest.raises(ConnectionError):
        call_with_retry("hi", attempts=3)
    assert len(calls) == 3

The last test fakes the network with monkeypatch, so it runs in milliseconds, costs nothing and never flakes. Test the behaviour you promised (it retries 3 times, then raises), not the implementation details.

the test pyramid: many fast unit tests with fakes at the bottom, fewer integration tests against real services in the middle, a handful of end-to-end checks on top.
Diagram— the test pyramid: many fast unit tests with fakes at the bottom, fewer integration tests against real services in the middle, a handful of end-to-end checks on top.

From production

A team's test suite calls the real LLM API. It takes 14 minutes, costs money on every run and fails randomly about 1 run in 10.

What's the most effective change?

  1. Run the tests less often
  2. Delete the flaky tests
  3. Retry failed tests until they pass
  4. Put the provider behind an interface and use fakes in unit tests; keep a small, separate set of real-API integration tests
Show the answer

D. Put the provider behind an interface and use fakes in unit tests; keep a small, separate set of real-API integration tests — Unit tests should be fast, free and deterministic, which means no network. A few integration tests still check the real provider, but they're separate and run less often. Retrying flaky tests hides real bugs.

4. Environments and packaging with uv

The question. How do you make "works on my machine" work on every machine?

Start from nothing. Each project needs its own virtual environment (an isolated set of installed packages) and a lockfile that pins every dependency to an exact version. The modern tool for both is uv: it creates the environment, resolves dependencies, writes uv.lock, and is much faster than pip. Project metadata lives in pyproject.toml.

bash
uv init atlas-tools          # creates pyproject.toml and a package skeleton
cd atlas-tools
uv add httpx pydantic        # adds dependencies and updates uv.lock
uv add --dev pytest mypy     # tools only needed for development
uv run pytest -q             # runs inside the project's environment
uv sync --frozen             # in CI or Docker: install exactly what the lockfile says

Commit both pyproject.toml and uv.lock. In CI and Docker, install with --frozen so a new release of some library can't silently change production. A typical layout puts code in src/atlas_tools/ and tests in tests/.

pyproject.toml (version ranges) → resolver (uv lock) → uv.lock (every package, exact) → CI/Docker (uv sync --frozen); below, without a lockfile a Friday build gets libX 1.9.4 and a Monday build gets a breaking 2.0.0, while with the lockfile every build is identical and upgrades arrive as tested PRs.
Diagram— pyproject.toml (version ranges) → resolver (uv lock) → uv.lock (every package, exact) → CI/Docker (uv sync --frozen); below, without a lockfile a Friday build gets libX 1.9.4 and a Monday build gets a breaking 2.0.0, while with the lockfile every build is identical and upgrades arrive as tested PRs.

Why AI projects feel this more than most. ML stacks have deep, fast-moving dependency trees. PyTorch builds are tied to CUDA versions, tokenizer libraries ship compiled wheels, and LLM SDKs release often. Three habits keep builds reproducible:

  • Pin the Python version too (requires-python in pyproject.toml, and a fixed base image in Docker). A minor Python upgrade can change library wheels and behaviour.
  • Separate runtime and development dependencies (uv add --dev), so production images don't ship test tools and linters.
  • Upgrade on a schedule, in their own PRs, with the test suite as the gate. Bots such as Dependabot and Renovate open these PRs automatically and can update lockfiles.

In production. A Docker build that copies pyproject.toml and uv.lock first, installs dependencies, and only then copies the source code lets Docker cache the slow dependency layer between builds. Code changes no longer reinstall PyTorch. The same lockfile is what makes "the eval ran on the same environment as production" a true statement. That matters when a metric moves and you need to know whether the model changed or a library did.

From production

Production breaks on Monday with no code changes. Investigation shows pip install -r requirements.txt pulled a new major version of a dependency over the weekend.

What prevents this?

  1. Run pip install twice
  2. A lockfile with exact versions (e.g. uv.lock), installed with --frozen in CI and Docker, and dependency upgrades done deliberately in their own PRs
  3. Use a global Python install on the server
  4. Pin nothing so you always get the latest fixes
Show the answer

B. A lockfile with exact versions (e.g. uv.lock), installed with --frozen in CI and Docker, and dependency upgrades done deliberately in their own PRs — Unpinned requirements resolve to whatever is newest at install time, so the same code can build differently on different days. A lockfile makes builds reproducible; upgrades happen on purpose and get tested.


Check yourself

  1. Why is except Exception: pass dangerous?
  2. What does raise X from err preserve?
  3. Name two advantages of logging over print.
  4. What problem does a lockfile solve?
  5. Why should an error response for an unexpected exception include its type but not its message?
  6. What makes a JSON log line more useful than a printed sentence during an incident?
Answers
  1. It silently hides every failure, including ones you never anticipated, so broken systems look healthy.
  2. The original exception as __cause__, so the traceback shows both the new error and what caused it.
  3. Levels you can filter per environment, timestamps and logger names, tracebacks via logger.exception, and configurable destinations.
  4. Builds that change over time: it pins every package (including indirect dependencies) to exact versions, so every install is identical.
  5. Unexpected errors can carry internal details (file paths, keys, SQL, other users' data) in their message. Log the full error internally, and return a generic message to the caller.
  6. Its fields can be filtered, grouped and graphed directly (latency by model, errors by request_id), instead of parsed with fragile text searches.

Exercise

Write load_config(text) that parses a JSON config string and returns a dict with model (str, required) and timeout (number, default 30, must be 1–120). Raise a custom ConfigError with a helpful message for invalid JSON, a missing model or a bad timeout, chaining the original error where there is one. Then write plain-assert tests for four cases.

Solution
python
import json

class ConfigError(Exception):
    pass

def load_config(text):
    try:
        data = json.loads(text)
    except json.JSONDecodeError as err:
        raise ConfigError(f"config is not valid JSON (line {err.lineno})") from err
    if not isinstance(data.get("model"), str):
        raise ConfigError("'model' is required and must be a string")
    timeout = data.get("timeout", 30)
    if not isinstance(timeout, (int, float)) or not 1 <= timeout <= 120:
        raise ConfigError(f"'timeout' must be between 1 and 120, got {timeout!r}")
    return {"model": data["model"], "timeout": timeout}

def expect_error(text, fragment):
    try:
        load_config(text)
    except ConfigError as err:
        assert fragment in str(err), err
        return err
    raise AssertionError("expected ConfigError")

assert load_config('{"model": "small"}') == {"model": "small", "timeout": 30}
err = expect_error('{"model": ', "not valid JSON")
assert isinstance(err.__cause__, json.JSONDecodeError)
expect_error('{"timeout": 10}', "'model' is required")
expect_error('{"model": "small", "timeout": 999}', "between 1 and 120")
print("all 4 tests passed")

Each failure message says what's wrong and what's expected, which is what an on-call engineer needs at 2 a.m. With pytest, expect_error becomes with pytest.raises(ConfigError, match=...).


This unlocks:

  • NumPy and pandas — fast numerical code, now with tests to prove it's still correct.
  • Profiling — finding the slow line before optimising anything.
  • Interview prep and capstone — a tested, packaged project you can show.