You already know: what makes kernels fast (coalescing, tiling, occupancy, the roofline: GPU memory hierarchy, Parallel computation, Performance engineering); PyTorch's tensor and autograd model (D4 PyTorch engineering).
Why this matters for Atlas. NVIDIA and the open-source community have spent decades tuning libraries for the operations deep learning uses most: matrix multiplication, convolution, attention, collectives, sorting and scanning. They reach performance that's hard to match, they're updated for each new GPU generation, and every framework already calls them. The engineering skill is knowing which library does what, how to feed it shapes it handles well, how to see which one is running in a profile, and recognizing the narrow cases where a custom kernel is worth writing (usually a fusion the libraries don't provide). This lesson is that map, with the shape rules quantified.
1. The map
The question. When PyTorch runs a model on a GPU, whose code is executing?
Start from nothing. Mostly libraries: