VFlash: Fast, Accurate Visual Planning via Flash Primitive Decoding
u_Unrolled
DSA for AI/ML · Module 13: Tries and strings

Lesson 13.6 of 6 in this module Unrolled Pro

One step of byte-pair encoding

Byte-pair encoding builds a subword vocabulary by repeatedly merging the most frequent adjacent pair of symbols. This lesson does one merge step correctly and efficiently: from rescanning the raw text for every occurrence (O(total characters) per step), to counting over distinct words weighted by frequency, to a pair-to-words index so a merge only rewrites the words that contain the pair and updates only their pair counts. Then tie-breaking, overlapping pairs like a a a, starting from bytes, and encoding new text with the learned merges in rank order.

about 15 min 3 Python examples 2 quick checks

Problem. A corpus is given as word frequencies. Each word starts as a sequence of single-character symbols. One BPE step:

  1. count every adjacent pair of symbols across the corpus, weighted by word frequency;
  2. pick the most frequent pair (ties: the lexicographically smallest pair);
  3. replace every occurrence of that pair, left to right, with one merged symbol.
text
words = {"learn": 5, "learner": 2, "learning": 6, "lean": 3, "earn": 4}
pair counts: (e, a) 20, (a, r) 17, (r, n) 17, (l, e) 16, ...
merge (e, a):  l ea r n   l ea r n e r   l ea r n i n g   l ea n   ea r n

Repeat the step thousands of times and frequent words become single symbols while rare words stay split into pieces: that's the subword vocabulary of most language models. Training runs many steps, so the cost of one step decides whether training takes minutes or days.

ML context. BPE (and its byte-level variant) is how GPT-style tokenizers are trained. Every merge adds one token to the vocabulary. The efficient step here, touching only the words that contain the merged pair, is the core of real tokenizer trainers; lesson 15.12 runs the full training loop with a priority queue over pair counts.


1. Step 1: Clarify

That's the preview. Modules 00–01 are free; this lesson is part of Unrolled Pro.
Staff AI & Pro Tier Lesson

Unlock "One step of byte-pair encoding"

The foundations are free forever. Unrolled Pro unlocks the complete lesson, production architectures, and 55 battle-tested ML system design case studies.

Everything included with Pro:
All 234+ Deep-Dive Lessons across transformer internals, inference, RAG & agents
55 Staff-Level ML & System Design Case Studies with mathematical capacity estimates
AI Engineer Sheet & Exercises with worked solutions, self-checks, and code
100 Landmark Research Papers guided walkthroughs & progress tracking
14-Day Money-Back Guarantee — 100% full refund if you aren't satisfied
View Plans & Unlock All Lessons

From ₹167/mo (~$2/mo) · Cancel anytime · Instant activation

100% Risk-Free: 14-day no-questions-asked money-back guarantee