Problem. A retrieval service embeds every incoming text (queries, chunks, user messages) with an embedding model. Each call costs about 20 ms of GPU time or an API fee, and real traffic repeats: the same popular queries, the same document chunks re-indexed, the same system prompts. Build a cache that:
- returns the stored vector for a text it has seen (a hit), and calls the model only for misses;
- accepts batches of texts, embedding all misses in one model call, each distinct text once;
- never uses more than a byte budget for stored vectors, evicting the least recently used ones;
- reports its hit rate.
budget = 300 KB, vectors of 96 float32 values (~0.4 KB each)
get_many(["what is rope?", "What is RoPE?", "kv cache"]) → 1 model call with 2 texts
get_many(["kv cache"]) → 0 model calls (hit)
Why it matters. Embedding the same text twice is pure waste, and at scale it's most of the traffic. Query distributions are heavily skewed (a few queries are very common), so a cache holding a small fraction of distinct texts answers most requests. But vectors are big (a 3,072-dimensional float32 vector is 12 KB), so the cache's limit must be in bytes, not entries.