VFlash: Fast, Accurate Visual Planning via Flash Primitive Decoding
u_Unrolled
Build Atlas

Part 1 of 9 · project

Data as tensors

Turn a folder of Markdown lessons into clean sections, question–answer pairs and batches of integers.

5 steps · about 10 min atlas/data.py scripts/part1_data.py

Before you start

This page builds the code. The ideas behind it are taught in the Library lessons listed at the end; read them first if a step uses something you haven't met. Inspect and copy the complete code in the Atlas Codebase Browser and run each step as you read.

Atlas starts as a folder of 22 Markdown files: the free lessons from this site. By the end of this part it can read them, cut them into sections with their heading paths, pull out every "Check yourself" question with its answer, and turn the whole corpus into arrays of integers that a model can train on. Every later part reads from what you build here.

Markdown files become clean text, then sections with heading paths, Q&A pairs, and finally integer ids split by position into train and validation.
Diagram— Markdown files become clean text, then sections with heading paths, Q&A pairs, and finally integer ids split by position into train and validation.

Step 1 · Clean the Markdown

A model should learn from prose, not from formatting. clean_markdown strips what isn't language: quiz blocks (they're JSON), code-fence markers, image placeholders, HTML tags, bold markers and link targets. The words of code examples stay, because Atlas should learn to talk about code.

python
def clean_markdown(md: str) -> str:
    md = re.sub(r"```quiz\n.*?```", "", md, flags=re.S)  # interactive quizzes are not prose
    md = re.sub(r"```[a-z]*( no-run)?\n", "", md).replace("```", "")
    md = re.sub(r"^> \*\*\[IMAGE[^\n]*$", "", md, flags=re.M)
    md = re.sub(r"<[^>]+>", "", md)
    md = re.sub(r"\*\*(.+?)\*\*", r"\1", md)
    md = re.sub(r"`([^`]*)`", r"\1", md)
    md = re.sub(r"\[([^\]]+)\]\([^)]+\)", r"\1", md)
    md = re.sub(r"\n{3,}", "\n\n", md)
    return md.strip()

Order matters: quizzes go before code fences, or the quiz JSON would survive as "code".

Step 2 · Split into sections, keeping the heading path

Part 2 classifies sections and Part 7 retrieves them, so a section needs to know where it came from. split_sections walks the lines, and every ## or ### heading updates a path like Dicts and Sets > 2. Hashing > Collisions.

python
def split_sections(md: str, title: str) -> list:
    out, path, buf = [], [title], []
    for line in md.splitlines():
        m = re.match(r"^(#{2,3}) (.+)$", line)
        if m:
            if buf and "".join(buf).strip():
                out.append((" > ".join(path), clean_markdown("\n".join(buf))))
            level = len(m.group(1))
            path = path[: level - 1] + [m.group(2).strip()]   # a ## resets any ### below it
            buf = []
        else:
            buf.append(line)
    if buf and "".join(buf).strip():
        out.append((" > ".join(path), clean_markdown("\n".join(buf))))
    return [(h, t) for h, t in out if len(t) > 40]

Sections shorter than 40 characters are dropped: they are usually a heading followed directly by a sub-heading.

Step 3 · Pull out questions and answers

Every lesson ends with numbered questions and, inside a <details> block, numbered answers in the same order. Pairing them gives Atlas 78 question–answer pairs: the fine-tuning data for Part 5 and the evaluation set for Parts 6 and 7.

python
def extract_qa(md: str) -> list:
    m = re.search(r"## Check yourself\n(.*?)<details><summary>Answers</summary>\n(.*?)</details>", md, re.S)
    if not m:
        return []
    def items(block: str) -> list:
        parts = re.split(r"^\d+\.\s+", block.strip(), flags=re.M)
        return [clean_markdown(p).replace("\n", " ").strip() for p in parts if p.strip()]
    qs, ans = items(m.group(1)), items(m.group(2))
    return [(q, a) for q, a in zip(qs, ans) if q and a]

Step 4 · Text to integers, split by position

Models eat numbers. For now each character gets an id (Part 4 replaces this with a real tokenizer). The split into training and validation is by position: the last 10% of the stream is validation. A random split would put neighbouring characters of the same sentence on both sides, and the validation loss would look better than the model really is.

python
def train_val_split(ids: np.ndarray, val_fraction: float = 0.1) -> tuple:
    n = int(len(ids) * (1 - val_fraction))
    return ids[:n], ids[n:]

Step 5 · Batches without a Python loop

Training wants random windows of block tokens, and the target for each position is simply the next token. Broadcasting builds a whole batch of windows in one indexing operation: starts[:, None] + offsets is a (batch, block) grid of positions.

python
def get_batch(ids: np.ndarray, block: int, batch: int, rng: np.random.Generator) -> tuple:
    starts = rng.integers(0, len(ids) - block - 1, size=batch)
    offsets = np.arange(block)
    x = ids[starts[:, None] + offsets]       # (batch, block) by broadcasting
    y = ids[starts[:, None] + offsets + 1]   # the same windows, shifted by one
    return x, y

y is x moved one step to the right: at every position the model will be asked to predict the character that comes next.

Run it

bash
python3 scripts/part1_data.py
text
22 lessons · 161 sections · 78 Q&A pairs · 179,218 characters
vocabulary 127 characters · train 161,296 · val 17,922 (split by position, not randomly)
most frequent: ' ' 17.7%, 'e' 8.1%, 't' 6.1%, 's' 5.3%, 'a' 4.9%, 'o' 4.8%, 'n' 4.8%, 'r' 4.5%
a batch: x (2, 16) y (2, 16) (y is x shifted by one: next-character targets)
x[0] = 'i is on a slow r'
y[0] = ' is on a slow re'

Read x[0] and y[0] together: every character in y is the one that follows the same position in x. That one-step shift is the whole training signal for the GPT in Part 4.

Check it works

bash
python3 -m pytest -q tests -k "corpus or clean or qa or sections or batches"

The tests check that the corpus loads with at least 60 Q&A pairs, that cleaning removes markup, that questions and answers pair up in order, that heading paths are kept, and that y == x + 1 on a counting sequence, with the validation split starting exactly where training ends.

Checkpoint. Atlas can load its documents and turn them into training batches. In Part 2 it learns something from them for the first time.

View Source Code

Go deeper in the Library

The lessons that teach the ideas this part uses, in the order to read them.