You already know: timeouts, retries and idempotency (D1 Networking, D9 Tools & protocols); transactions and isolation on one database (D1 Databases); journals and effectively-once side effects (D9 Advanced agent engineering); the D13 bridge's three replicas that drifted apart.
Why this matters for Atlas. Every production AI system is distributed: model servers, feature stores, vector databases, queues, training clusters. Distribution brings partial failure (some parts fail while others keep running), unreliable networks (messages delayed, lost, duplicated), and no shared clock. Under those conditions, simple questions ("did the write succeed?", "which update came first?", "do all replicas agree?") need careful answers. This lesson builds the core vocabulary and algorithms from first principles: failure models, logical time, consistency models and the trade-offs between them, quorums, and consensus with Raft. Everything else in this domain, and the distributed training of Domain 15, stands on it.
1. What can go wrong: failure models
The question. A request to another server times out. What happened?
Start from nothing. You can't tell. The request may have been lost, the server may have crashed before or after doing the work, the server may be slow, or the reply may have been lost. From the caller's side, a slow node and a dead node look identical. That's the defining difficulty of distributed systems, and the models below say which failures a design must survive: