VFlash: Fast, Accurate Visual Planning via Flash Primitive Decoding
u_Unrolled
D8: RAG & Knowledge Systems · Retrieval I — The Pipeline, the Inverted Index and BM25 from First Principles
Phase 5/Domain 8/Lesson 1

Retrieval I — The Pipeline, the Inverted Index and BM25 from First Principles

25 min read·Advanced· Unrolled Pro
Retrieval I — The Pipeline, the Inverted Index and BM25 from First Principles

You already know: hash maps and sorting (D1 Data structures); TF-IDF (D3 Feature engineering); context budgets and delimiting sources (D7 Context engineering, Prompt fundamentals).

Why this matters for Atlas. Retrieval-augmented generation (RAG) is the pattern behind almost every assistant that answers from a company's own knowledge: search first, then let the model answer from what was found, citing it. The model's answer can only be as good as what retrieval hands it. If the right passage isn't retrieved, the best model in the world will either guess or say it doesn't know. This lesson builds retrieval from the ground up — the inverted index and BM25, which still power most search engines — and measures exactly where word matching breaks, which motivates everything else in this domain. Lesson II (Retrieval II) takes it to production: analyzers, knowing when not to answer, speed, sharding, permissions and debugging.

Where you'll use this. Every RAG system has a retrieval step, and most production ones still include BM25: in Elasticsearch, OpenSearch, Postgres full-text search or a library like Tantivy. Knowing how it scores tells you why a query found the wrong passage, when keyword search will beat embeddings (codes, names, error messages), and what the dense retrievers in the rest of this domain are trying to improve on.


1. The RAG pipeline

The question. What actually happens between a user's question and a grounded answer?

Start from nothing. Two phases:

That's the preview. The full lesson, with its code and exercises, is part of Unrolled Pro.
Staff AI & Pro Tier Lesson

Unlock "Retrieval I — The Pipeline, the Inverted Index and BM25 from First Principles"

The foundations are free forever. Unrolled Pro unlocks the complete lesson, production architectures, and 55 battle-tested ML system design case studies.

Everything included with Pro:
All 234+ Deep-Dive Lessons across transformer internals, inference, RAG & agents
55 Staff-Level ML & System Design Case Studies with mathematical capacity estimates
AI Engineer Sheet & Exercises with worked solutions, self-checks, and code
100 Landmark Research Papers guided walkthroughs & progress tracking
14-Day Money-Back Guarantee — 100% full refund if you aren't satisfied
View Plans & Unlock All Lessons

From ₹167/mo (~$2/mo) · Cancel anytime · Instant activation

100% Risk-Free: 14-day no-questions-asked money-back guarantee
Build Atlas · Part 7 of 9: Ground it in your documentsUse this lesson in the project: Chunk the lessons by structure, search them with BM25 and dense vectors, fuse the rankings, and measure retrieval apart from answers.