VFlash: Fast, Accurate Visual Planning via Flash Primitive Decoding
u_Unrolled
DSA for AI/ML · Module 07: Linked lists and caches

Lesson 7.8 of 8 in this module Unrolled Pro

Cache Recent Embeddings with Eviction

Put an LRU cache in front of an embedding model so repeated texts are never embedded twice, under a fixed memory budget. From calling the model every time, to an unbounded dict that grows until the process dies, to an LRU capped by entry count (which breaks when vectors differ in size), to an LRU capped by bytes with batched, de-duplicated misses. Then model upgrades and cache keys, hashing and normalising texts, sharing a cache across processes, and why a 'semantic' cache for near-duplicates is a different, riskier tool.

about 15 min 4 Python examples 2 quick checks

Problem. A retrieval service embeds every incoming text (queries, chunks, user messages) with an embedding model. Each call costs about 20 ms of GPU time or an API fee, and real traffic repeats: the same popular queries, the same document chunks re-indexed, the same system prompts. Build a cache that:

  • returns the stored vector for a text it has seen (a hit), and calls the model only for misses;
  • accepts batches of texts, embedding all misses in one model call, each distinct text once;
  • never uses more than a byte budget for stored vectors, evicting the least recently used ones;
  • reports its hit rate.
text
budget = 300 KB, vectors of 96 float32 values (~0.4 KB each)
get_many(["what is rope?", "What is  RoPE?", "kv cache"])   → 1 model call with 2 texts
get_many(["kv cache"])                                        → 0 model calls (hit)

Why it matters. Embedding the same text twice is pure waste, and at scale it's most of the traffic. Query distributions are heavily skewed (a few queries are very common), so a cache holding a small fraction of distinct texts answers most requests. But vectors are big (a 3,072-dimensional float32 vector is 12 KB), so the cache's limit must be in bytes, not entries.


1. Step 1: Clarify

That's the preview. Modules 00–01 are free; this lesson is part of Unrolled Pro.
Staff AI & Pro Tier Lesson

Unlock "Cache Recent Embeddings with Eviction"

The foundations are free forever. Unrolled Pro unlocks the complete lesson, production architectures, and 55 battle-tested ML system design case studies.

Everything included with Pro:
All 234+ Deep-Dive Lessons across transformer internals, inference, RAG & agents
55 Staff-Level ML & System Design Case Studies with mathematical capacity estimates
AI Engineer Sheet & Exercises with worked solutions, self-checks, and code
100 Landmark Research Papers guided walkthroughs & progress tracking
14-Day Money-Back Guarantee — 100% full refund if you aren't satisfied
View Plans & Unlock All Lessons

From ₹167/mo (~$2/mo) · Cancel anytime · Instant activation

100% Risk-Free: 14-day no-questions-asked money-back guarantee