VFlash: Fast, Accurate Visual Planning via Flash Primitive Decoding
u_Unrolled
All projects

Inference · Advanced · by Andrej Karpathy

llm.c: LLM training in pure C/CUDA

Simple, pure C and CUDA implementation for training GPT-2 and GPT-3 without 245MB of PyTorch runtime overhead.

2–3 weeks 26800 on GitHubCCUDAcuBLASC11
Open the code on GitHub

What you'll learn

  • Manual forward and backward pass implementation in pure C
  • Writing raw CUDA kernels for multi-head attention and LayerNorm
  • Multi-GPU data parallel training using MPI
  • Exact numerical matching against PyTorch autograd outputs

Read first

The lessons that explain what this project uses.

Part of the Inference Engineering path.

Milestones

0/4

Built it? Show it.

Add your own repo to earn 150 XP and the Shipper badge. It appears on your profile, so it's ready for your resume.

More inference projects