You already know: NumPy arrays are packed memory (D1 Python internals §3); row-major layout and cache lines (D1 C++ §5); shared memory between processes (D1 OS §6); autograd as reverse-mode differentiation (D2 Calculus §4); the building blocks and architectures of neural networks (this domain).
Why this matters for Atlas. The difference between a model that trains in a day and one that takes a week — or silently trains wrong — is usually engineering: unnecessary copies, a broken autograd graph, a data loader starving the GPU, memory blown on activations, or a checkpoint that can't be loaded safely. This lesson covers the PyTorch mechanics that every LLM training and serving stack relies on.
Examples run on CPU. GPU-only lines are marked; on a GPU, move tensors with
.to("cuda").
1. Tensor memory: storage, strides, views and copies
The question. x.T on a 10 GB tensor returns instantly. How is that possible, and when does PyTorch secretly copy?
Start from nothing. A tensor is a view onto a flat block of memory (its storage), described by: