VFlash: Fast, Accurate Visual Planning via Flash Primitive Decoding
u_Unrolled
D13: GPU & CUDA Programming · CUDA Libraries & Tools — cuBLAS, cuDNN, CUTLASS, NCCL and When Not to Write a Kernel

CUDA Libraries & Tools — cuBLAS, cuDNN, CUTLASS, NCCL and When Not to Write a Kernel

15 min read·Staff+ Tier· Unrolled Pro
CUDA Libraries & Tools — cuBLAS, cuDNN, CUTLASS, NCCL and When Not to Write a Kernel

You already know: what makes kernels fast (coalescing, tiling, occupancy, the roofline: GPU memory hierarchy, Parallel computation, Performance engineering); PyTorch's tensor and autograd model (D4 PyTorch engineering).

Why this matters for Atlas. NVIDIA and the open-source community have spent decades tuning libraries for the operations deep learning uses most: matrix multiplication, convolution, attention, collectives, sorting and scanning. They reach performance that's hard to match, they're updated for each new GPU generation, and every framework already calls them. The engineering skill is knowing which library does what, how to feed it shapes it handles well, how to see which one is running in a profile, and recognizing the narrow cases where a custom kernel is worth writing (usually a fusion the libraries don't provide). This lesson is that map, with the shape rules quantified.


1. The map

The question. When PyTorch runs a model on a GPU, whose code is executing?

Start from nothing. Mostly libraries:

That's the preview. The full lesson, with its code and exercises, is part of Unrolled Pro.
Staff AI & Pro Tier Lesson

Unlock "CUDA Libraries & Tools — cuBLAS, cuDNN, CUTLASS, NCCL and When Not to Write a Kernel"

The foundations are free forever. Unrolled Pro unlocks the complete lesson, production architectures, and 55 battle-tested ML system design case studies.

Everything included with Pro:
All 234+ Deep-Dive Lessons across transformer internals, inference, RAG & agents
55 Staff-Level ML & System Design Case Studies with mathematical capacity estimates
AI Engineer Sheet & Exercises with worked solutions, self-checks, and code
100 Landmark Research Papers guided walkthroughs & progress tracking
14-Day Money-Back Guarantee — 100% full refund if you aren't satisfied
View Plans & Unlock All Lessons

From ₹167/mo (~$2/mo) · Cancel anytime · Instant activation

100% Risk-Free: 14-day no-questions-asked money-back guarantee