VFlash: Fast, Accurate Visual Planning via Flash Primitive Decoding
u_Unrolled
DSA for AI/ML · Module 09: Heaps and top-k

Lesson 9.9 of 12 in this module Unrolled Pro

Top-k Tokens from a Logits Array

Pick the k most likely next tokens from a model's logits and sample among them. From a softmax over the whole vocabulary followed by a full sort (O(V log V)), to a heap of size k over the raw logits, since softmax preserves order (O(V log k)), to a linear-time selection of the k-th largest logit as a threshold. Then nucleus (top-p) sampling with a heap that's popped only until the probability mass is reached, temperature, the overflow that a naive softmax hits and the max-subtraction that fixes it, and checking that the sampler's frequencies match its probabilities.

about 15 min 6 Python examples 2 quick checks

Problem. A language model outputs a logit for every token in its vocabulary of V tokens (V ≈ 50,000 to 250,000). Top-k sampling keeps the k tokens with the highest logits, turns their logits into probabilities with a softmax over those k only, and samples the next token from them.

text
logits = [2.0, -1.0, 0.5, 3.0, 1.0],  k = 2
top-2 tokens: 3 (logit 3.0), 0 (logit 2.0)
probabilities: softmax([3.0, 2.0]) = [0.731, 0.269]

This runs once per generated token, for every sequence in the batch. Generating 500 tokens for 64 sequences means 32,000 selections over 50,000+ logits: the per-call cost matters.

Why it matters. Top-k (and its cousin top-p) is how most chat models avoid both the blandness of greedy decoding and the nonsense of sampling from the full long tail. The selection step is a heap or selection problem; the softmax step is a numerical-stability problem. Both are classic interview follow-ups for ML roles.


1. Step 1: Clarify

That's the preview. Modules 00–01 are free; this lesson is part of Unrolled Pro.
Staff AI & Pro Tier Lesson

Unlock "Top-k Tokens from a Logits Array"

The foundations are free forever. Unrolled Pro unlocks the complete lesson, production architectures, and 55 battle-tested ML system design case studies.

Everything included with Pro:
All 234+ Deep-Dive Lessons across transformer internals, inference, RAG & agents
55 Staff-Level ML & System Design Case Studies with mathematical capacity estimates
AI Engineer Sheet & Exercises with worked solutions, self-checks, and code
100 Landmark Research Papers guided walkthroughs & progress tracking
14-Day Money-Back Guarantee — 100% full refund if you aren't satisfied
View Plans & Unlock All Lessons

From ₹167/mo (~$2/mo) · Cancel anytime · Instant activation

100% Risk-Free: 14-day no-questions-asked money-back guarantee