You already know: partial failure, timeouts and idempotency (Distributed systems fundamentals); replication and failover (Distributed storage); retries with backoff, rate limits and load balancing (D1 Networking); monitoring and alerting (D11 Model monitoring); capacity cliffs in LLM serving (D12 Measurement & capacity).
Why this matters for Atlas. Reliability isn't "never fail": every component fails eventually, and at scale something is always failing. Site reliability engineering turns reliability into an engineering quantity: define what users need (SLIs and SLOs), spend the allowed unreliability deliberately (error budgets), build redundancy whose numbers you can compute, keep overload from turning into collapse (retry storms, load shedding, circuit breakers), degrade gracefully (fallback models), and learn from incidents. AI systems add their own twists: GPU capacity is scarce and slow to add, model dependencies are heavy, and "up but wrong" is a real failure mode. This lesson gives the numbers and mechanisms behind each.
1. SLIs, SLOs and error budgets
The question. What does "99.9% uptime" mean for Atlas's tutor?
Start from nothing. Reliability is defined from the user's side: