Task. Given a corpus (a list of texts), build a vocabulary:
- reserved special tokens first:
<pad>→ 0,<unk>→ 1; - then every token that appears at least
min_counttimes, ordered by frequency (descending), ties broken alphabetically, at mostmax_sizetokens in total; - plus
encode(text) → ids(unknown tokens →<unk>) anddecode(ids) → tokens.
corpus = ["the cat sat", "the cat ran", "a dog sat"]
min_count = 2
vocab → {"<pad>": 0, "<unk>": 1, "cat": 2, "sat": 3, "the": 4}
encode("the dog sat") → [4, 1, 3]
ML context. This is the ML task: every word-level or subword tokenizer starts from counting. The decisions here (how to count fast, how to keep ids deterministic, what to do with rare and unknown tokens, what to do when counts don't fit in memory) are real engineering decisions in training pipelines.