You already know: BM25 and inverted indexes (Retrieval fundamentals); embeddings and the cost of embedding text (Embeddings); data cleaning and deduplication (D5 LLM pretraining §1); hashing (D1).
Why this matters for Atlas. "Garbage in, garbage out" is truer for RAG than almost anywhere else. If a navigation menu gets indexed, it matches every query. If a fact is split across two chunks, neither chunk answers the question. If a chunk doesn't record which section it came from, "It costs 249 rupees" is useless. And if a deleted policy stays in the index, the assistant confidently cites rules that no longer exist. Most RAG quality problems teams blame on "the model" are really ingestion problems. This lesson builds each stage and measures the decisions that matter.
1. Parsing: keep the content, drop the chrome, keep the structure
The question. A web page is mostly menus, scripts, banners and footers. How do you extract just the content, without losing its structure?
Start from nothing. Walk the HTML and: