Problem. You have N tokenised examples (lists of token IDs of different lengths) with labels. Each epoch, produce mini-batches of
B examples in a fresh random order (reproducible from a seed). Each batch is a tokens array (B × L) padded with pad_id, a boolean
mask (B × L) marking real tokens, and the labels. Every example appears exactly once per epoch.
seqs = [[5, 3, 9], [7], [2, 8], [4, 4, 4, 4]], B = 2, pad_id = 0
one batch: tokens = [[2, 8, 0, 0], mask = [[1, 1, 0, 0],
[4, 4, 4, 4]] [1, 1, 1, 1]]
The model's cost is proportional to B × L including padding. So the real optimisation target isn't Python speed; it's how many padding tokens you feed the GPU. Shuffling (for unbiased gradient estimates) fights against grouping similar lengths (for less padding); bucketing (Step 3) gets most of both.
ML context. This is the data loader of every sequence model, and a frequent ML-round question because the details matter: a missing mask lets attention read padding, a shuffle without a seed breaks reproducibility, and a fixed batch size with long-tailed lengths wastes most of the GPU. Token-budget batching (Change A) is how large-model training keeps memory constant per step.