VFlash: Fast, Accurate Visual Planning via Flash Primitive Decoding
u_Unrolled
D13: GPU & CUDA Programming · Triton Programming — Fast Fused Kernels in Python

Triton Programming — Fast Fused Kernels in Python

20 min read·Staff+ Tier· Unrolled Pro
Triton Programming — Fast Fused Kernels in Python

You already know: the CUDA model of grids, blocks and threads (CUDA programming model); coalescing, shared memory and tiling (GPU memory hierarchy); reductions, online softmax and tiled matmul (Parallel computation); when a custom kernel is worth writing (CUDA libraries & tools); FlashAttention's tiling (D12 Advanced optimization).

Why this matters for Atlas. Triton (OpenAI, now an open-source compiler that PyTorch 2 uses internally) lets you write GPU kernels in Python at the level of blocks of data instead of individual threads. You say "this program instance loads a 128-element block, computes, stores it"; the compiler decides how threads map to elements, coalesces memory accesses, stages data through shared memory and uses tensor cores. Fused kernels that take weeks in CUDA take hours, at performance often within 10% of hand-tuned code. This lesson teaches Triton's programming model and builds four real kernels, each paired with a NumPy emulation of the exact same block program so you can run and verify the logic without a GPU.


1. The programming model: programs over blocks

The question. What does a Triton kernel describe, if not a thread?

Start from nothing. A Triton kernel is run as a grid of program instances. Each instance gets an ID (tl.program_id(axis)) and works on a block of data, a vector or tile whose size (BLOCK_SIZE) is a compile-time constant (a power of two). Inside, you write array operations on whole blocks: tl.load a block of pointers (with a mask for out-of-range elements), compute with +, *, tl.exp, tl.sum, tl.max, tl.dot, then tl.store the result. There's no thread index, no __syncthreads, no explicit shared memory: the compiler handles them.

That's the preview. The full lesson, with its code and exercises, is part of Unrolled Pro.
Staff AI & Pro Tier Lesson

Unlock "Triton Programming — Fast Fused Kernels in Python"

The foundations are free forever. Unrolled Pro unlocks the complete lesson, production architectures, and 55 battle-tested ML system design case studies.

Everything included with Pro:
All 234+ Deep-Dive Lessons across transformer internals, inference, RAG & agents
55 Staff-Level ML & System Design Case Studies with mathematical capacity estimates
AI Engineer Sheet & Exercises with worked solutions, self-checks, and code
100 Landmark Research Papers guided walkthroughs & progress tracking
14-Day Money-Back Guarantee — 100% full refund if you aren't satisfied
View Plans & Unlock All Lessons

From ₹167/mo (~$2/mo) · Cancel anytime · Instant activation

100% Risk-Free: 14-day no-questions-asked money-back guarantee