VFlash: Fast, Accurate Visual Planning via Flash Primitive Decoding
u_Unrolled
DSA for AI/ML · Module 15: The ML coding round

Lesson 15.12 of 12 in this module Unrolled Pro

Byte-pair encoding merges

Train a byte-pair encoding tokenizer: learn M merges by repeatedly merging the most frequent adjacent pair. From recounting every pair in the corpus after each merge (O(M · S) for S symbols), to lesson 13.6's incremental update that rewrites only words containing the merged pair, plus a max-heap of pair counts with lazy deletion so the best pair is found in O(log P) instead of scanning all pairs. Then byte-level training with an exact round trip, encoding with merge ranks, pre-tokenisation and special tokens, and a target vocabulary size.

about 15 min 3 Python examples 2 quick checks

Problem. Given a corpus as word frequencies, learn M merges:

text
repeat M times:
    count every adjacent symbol pair, weighted by word frequency
    pick the most frequent pair (ties: the smallest pair)
    merge it everywhere, left to right, and record it

Return the ordered list of merges. Lesson 13.6 did one step efficiently. Training does tens of thousands, so two costs dominate: rewriting the corpus (solved in 13.6: touch only the words containing the pair) and finding the best pair each step, which a scan over all P distinct pairs makes O(P) per merge. A heap fixes the second.

ML context. This is how GPT-style tokenizers are trained, usually on bytes (Change A) so any text can be encoded. Vocabulary size (the number of merges plus 256) is a model design choice: larger vocabularies make sequences shorter but embedding tables bigger. Implementing the trainer is a "hard" ML-round question because it combines a hash map, an inverted index and a priority queue with lazy deletion.


1. Step 1: Clarify

That's the preview. Modules 00–01 are free; this lesson is part of Unrolled Pro.
Staff AI & Pro Tier Lesson

Unlock "Byte-pair encoding merges"

The foundations are free forever. Unrolled Pro unlocks the complete lesson, production architectures, and 55 battle-tested ML system design case studies.

Everything included with Pro:
All 234+ Deep-Dive Lessons across transformer internals, inference, RAG & agents
55 Staff-Level ML & System Design Case Studies with mathematical capacity estimates
AI Engineer Sheet & Exercises with worked solutions, self-checks, and code
100 Landmark Research Papers guided walkthroughs & progress tracking
14-Day Money-Back Guarantee — 100% full refund if you aren't satisfied
View Plans & Unlock All Lessons

From ₹167/mo (~$2/mo) · Cancel anytime · Instant activation

100% Risk-Free: 14-day no-questions-asked money-back guarantee