VFlash: Fast, Accurate Visual Planning via Flash Primitive Decoding
u_Unrolled
DSA for AI/ML · Module 15: The ML coding round

Lesson 15.9 of 12 in this module Unrolled Pro

Top-k and top-p sampling

Sample the next token from a model's logits after applying temperature, keeping only the k most likely tokens (top-k) or the smallest set whose probability reaches p (top-p, nucleus), and renormalising. From sorting the whole vocabulary and sampling with a Python loop over the cumulative distribution (O(V log V) and O(V) interpreted per token), to argpartition for top-k (O(V)), a sort of only the candidates for top-p, a stable softmax, and one searchsorted to draw the sample, for a whole batch at once. Then greedy decoding at temperature 0, min-p, repetition penalties, and reproducible sampling.

about 10 min 3 Python examples 2 quick checks

Problem. Given a row of logits over a vocabulary of V tokens, a temperature T, and either k (top-k) or p (top-p), draw one token:

  1. divide the logits by T and apply softmax;
  2. top-k: keep the k most likely tokens; top-p: keep the smallest set of most-likely tokens whose probabilities sum to at least p (always including the token that crosses p);
  3. renormalise the kept probabilities and sample one token from them.
text
probs (after temperature) = [0.50, 0.20, 0.15, 0.10, 0.05]   (tokens 0..4)
top-k, k = 2   →  sample from {0: 0.714, 1: 0.286}
top-p, p = 0.8 →  0.50 + 0.20 = 0.70 < 0.8, + 0.15 = 0.85 ≥ 0.8  →  sample from {0, 1, 2}, renormalised

V is 32,000 to 250,000 and this runs once per generated token, for every sequence in the batch. Sorting the entire vocabulary per token, then walking a Python loop to sample, is the brute force. The optimisations: select instead of sort, sort only what's left, and draw the sample with one binary search.

ML context. This is the decoding step of every LLM sampling API: temperature, top_k and top_p are the knobs. Getting the edge cases right matters in production: the token that crosses p must be kept (or top-p can return nothing), temperature 0 must mean greedy, and sampling must be reproducible from a seed for evaluation.


1. Step 1: Clarify

That's the preview. Modules 00–01 are free; this lesson is part of Unrolled Pro.
Staff AI & Pro Tier Lesson

Unlock "Top-k and top-p sampling"

The foundations are free forever. Unrolled Pro unlocks the complete lesson, production architectures, and 55 battle-tested ML system design case studies.

Everything included with Pro:
All 234+ Deep-Dive Lessons across transformer internals, inference, RAG & agents
55 Staff-Level ML & System Design Case Studies with mathematical capacity estimates
AI Engineer Sheet & Exercises with worked solutions, self-checks, and code
100 Landmark Research Papers guided walkthroughs & progress tracking
14-Day Money-Back Guarantee — 100% full refund if you aren't satisfied
View Plans & Unlock All Lessons

From ₹167/mo (~$2/mo) · Cancel anytime · Instant activation

100% Risk-Free: 14-day no-questions-asked money-back guarantee