Task. Given a sequence of tokens, find the adjacent pair (a, b) that occurs most often, breaking ties by
first appearance. Then merge every occurrence of that pair into one token a + b, scanning left to right
without overlaps.
tokens = ["l", "o", "w", "l", "o", "w", "e", "r"]
pairs: (l,o)×2 (o,w)×2 (w,l)×1 (w,e)×1 (e,r)×1
most frequent (first seen on a tie): (l, o)
merged → ["lo", "w", "lo", "w", "e", "r"]
ML context. This is one step of byte-pair encoding (BPE), the algorithm behind the subword tokenizers of most modern language models. Training BPE repeats "count pairs, merge the best" thousands of times, so how you count, and how you avoid recounting, decides whether training a tokenizer takes seconds or hours.