You already know: prompt anatomy and delimiting (Prompt fundamentals); attention and the KV cache (D5 Transformer architecture); TF-IDF similarity (D3); LRU caches and hashing (D1 Data structures).
Why this matters for Atlas. Prompt engineering is about how you ask. Context engineering is about what the model sees when you ask: which instructions, documents, memories, tool definitions and conversation turns go into each request, in what order, and how they're kept fresh and cheap. For anything beyond a single question — a tutor that remembers a student, an assistant over a knowledge base, an agent running for an hour — context is where quality, latency and cost are won or lost. Even with million-token windows, filling the window is rarely the answer.
1. The context window is a budget
The question. If a model accepts a million tokens, why not send everything?
Start from nothing. Three reasons:
- Cost and latency scale with input. Every input token is processed on every request (D5). A 500k-token prompt costs about 1,000× a 500-token one and takes far longer before the first output token.
- Attention gets diluted. Models use information at the start and end of a long context more reliably than information buried in the middle (Lost in the Middle, Liu et al., 2023). Newer models handle long contexts much better, but irrelevant material still competes with relevant material for attention.
- Irrelevant context causes errors. Distractors that look relevant can mislead a model. Long, stale tool outputs make agents lose track of the task.
So treat the window as a budget with line items: system prompt and tool definitions (stable), memory (durable facts about the user), retrieved documents (per request), conversation history (growing), and a reserve for the output. The job is to fill it with the highest-value tokens:
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.metrics.pairwise import cosine_similarity
approx_tokens = lambda text: max(1, len(text) // 4)
# Atlas's knowledge base, already split into chunks (D8 covers how to split well)
chunks = [
"The KV cache stores keys and values for every previous token so attention does not recompute them.",
"KV cache memory grows linearly with sequence length, layers, KV heads and head dimension.",
"Grouped-query attention shares each key/value head across several query heads, shrinking the KV cache.",
"PagedAttention stores the KV cache in fixed-size blocks, like virtual memory pages, to avoid fragmentation.",
"LoRA freezes the base weights and trains two small low-rank matrices per adapted layer.",
"Continuous batching adds new requests to a running batch at every decoding step.",
"Quantizing the KV cache to 8 bits halves its memory with a small quality cost.",
"Refunds for the yearly plan are available within 7 days of purchase.",
"Speculative decoding drafts several tokens with a small model and verifies them in one pass.",
"RMSNorm rescales activations by their root mean square, without subtracting the mean.",
] * 3 # a bigger corpus than the budget can hold
chunks = [f"{c} (copy {i // 10 + 1})" if i >= 10 else c for i, c in enumerate(chunks)]
def pack(query, budget_tokens, min_score=0.1):
"""Score chunks against the query; keep the best ones that fit, skip near-duplicates."""
vec = TfidfVectorizer(stop_words="english").fit(chunks + [query])
scores = cosine_similarity(vec.transform([query]), vec.transform(chunks))[0]
picked, used, seen, skipped = [], 0, set(), {"irrelevant": 0, "duplicate": 0, "over budget": 0}
for i in sorted(range(len(chunks)), key=lambda i: -scores[i]):
base = chunks[i].split(" (copy")[0]
cost = approx_tokens(chunks[i])
reason = ("irrelevant" if scores[i] < min_score else "duplicate" if base in seen
else "over budget" if used + cost > budget_tokens else None)
if reason:
skipped[reason] += 1
continue
picked.append((scores[i], chunks[i])); used += cost; seen.add(base)
return picked, used, skipped
def edges_first(items):
"""Put the strongest items at the start and end; the weakest end up in the middle."""
front, back = [], []
for k, item in enumerate(items): # items arrive sorted best-first
(front if k % 2 == 0 else back).append(item)
return front + back[::-1]
window, reserved_output, system_and_tools = 4_000, 1_000, 2_920
budget = window - reserved_output - system_and_tools
picked, used, skipped = pack("How can I reduce KV cache memory?", budget)
print(f"context window {window} = system+tools {system_and_tools} + output reserve {reserved_output} "
f"+ retrieval budget {budget}")
print(f"packed {len(picked)} chunks ({used} tokens) out of {len(chunks)} | skipped: {skipped}")
for score, text in edges_first(picked):
print(f" {score:.2f} {text}")
Out of 30 candidate chunks, the packer keeps the 3 most relevant that fit the 80-token budget (the numbers are small so you can see the decisions). It skips 15 as irrelevant, 6 as duplicates of chunks already taken, and 6 because they didn't fit. edges_first then orders them so the strongest chunks sit at the start and end of the block, and the weakest in the middle, where attention is least reliable.
Engineering reality. Retrieval quality dominates everything else here. TF-IDF misses synonyms ("shrink" vs "reduce"), which is why D8 uses embeddings, hybrid search and re-ranking. Deduplication matters more than it looks: knowledge bases are full of near-copies, and three versions of one paragraph waste budget and over-weight that source.
2. Memory: what to keep from a long conversation
The question. A tutoring session runs for 60 turns. What should the model still see at turn 59?
Start from nothing. Three common strategies:
- Full history: perfect recall, but cost grows every turn until it overflows the window.
- Sliding window: keep the last N turns. Bounded cost, but anything older is forgotten.
- Summary + window + pinned facts: replace old turns with a short summary, keep recent turns verbatim, and store durable facts (the student's goal, their level, their preferences) explicitly, so they never fall out.
approx_tokens = lambda text: max(1, len(text) // 4)
# A 60-turn tutoring conversation. Turn 3 states a fact the tutor must remember later.
turns = []
for t in range(60):
if t == 3:
user = "By the way, I'm preparing for a GPU-infrastructure interview at a big AI lab next month."
else:
user = f"Question {t}: can you explain step {t} of the KV-cache paging example in more detail?"
turns.append((user, f"Answer {t}: " + "Here is a detailed explanation of that step. " * 6))
def full_history(t):
return [m for pair in turns[: t + 1] for m in pair]
def sliding_window(t, keep=6):
return [m for pair in turns[max(0, t + 1 - keep): t + 1] for m in pair]
def summary_plus_window(t, keep=6, pinned=()):
"""Older turns are replaced by a short summary; durable facts are pinned explicitly."""
old = max(0, t + 1 - keep)
summary = f"[Summary of turns 0-{old - 1}: student is working through the KV-cache paging example.]" if old else ""
return [summary, *pinned] + [m for pair in turns[old: t + 1] for m in pair]
pinned = ["[Pinned fact: student is preparing for a GPU-infrastructure interview at an AI lab.]"]
strategies = {
"full history": full_history,
"sliding window (6 turns)": sliding_window,
"summary + window + pinned facts": lambda t: summary_plus_window(t, pinned=pinned),
}
print(f"{'strategy':<34}{'tokens at turn 10':>18}{'at turn 59':>12}{'remembers interview?':>22}")
for name, build in strategies.items():
at10 = sum(approx_tokens(m) for m in build(10))
ctx59 = build(59)
at59 = sum(approx_tokens(m) for m in ctx59)
remembers = any("interview" in m for m in ctx59)
print(f"{name:<34}{at10:>18,}{at59:>12,}{str(remembers):>22}")
| Strategy | Tokens at turn 10 | Tokens at turn 59 | Remembers the interview (said at turn 3)? |
|---|---|---|---|
| Full history | 992 | 5,402 | Yes |
| Sliding window (6 turns) | 540 | 540 | No |
| Summary + window + pinned facts | 580 | 581 | Yes |
The summary-and-pinned strategy keeps cost flat and keeps the fact that matters. The hard part in production is deciding what to pin or summarize. Usually a model does it, with instructions on what counts as durable ("goals, constraints, decisions, user preferences — not small talk").
Engineering reality. APIs now offer this server-side. Compaction summarizes earlier context automatically when a conversation approaches a size threshold. Context editing clears old tool results (often the bulk of an agent's context) once they've been used. For memory that must survive across sessions — "this student prefers Python examples" — store facts in a database and retrieve them, which is exactly the RAG problem of D8 applied to your own users. D9 (Agent memory) goes deeper.
3. Ordering for the cache: stable first, volatile last
The question. Most of every request (system prompt, tools, earlier turns) is identical to the previous request. Do we really pay to process it again each time?
Start from nothing. Not if the provider caches it. Prompt caching stores the model's processed state (the KV cache from D5) for a request prefix. When a later request starts with exactly the same tokens, that prefix is read from cache: much faster and much cheaper (on the Claude API, cache reads cost about 0.1× the normal input price and cache writes about 1.25×). The key word is prefix: the cache matches from the very first token, and any change invalidates everything after it. So the order of your context matters:
import hashlib
PRICE_INPUT = 4.00 / 1e6 # $ per input token (Claude Opus 5.5 list price, Nov 2026)
WRITE, READ = 1.25, 0.10 # cache write / cache read multipliers on the input price
SYSTEM = "You are Atlas, a tutor... " + "rules and lesson notes " * 3000 # ~18k-token stable prefix
TOOLS = "tool schemas: search_lessons, run_python, check_answer " * 400 # ~5k tokens
approx_tokens = lambda text: len(text) // 4
def blocks(layout, turn, history):
timestamp = f"Current time: 2026-11-11 10:{turn:02d}"
if layout == "timestamp first": # a volatile line at the very top changes every request
return [timestamp, TOOLS, SYSTEM] + history
return [TOOLS, SYSTEM] + history + [timestamp] # stable first, volatile last
def simulate(layout, turns=20):
cache, total, hit_tokens, uncached = set(), 0.0, 0, 0.0
history = []
for turn in range(turns):
history = history + [f"user {turn}: question about lesson {turn}", f"assistant {turn}: " + "answer " * 150]
parts = blocks(layout, turn, history)
prefix_hash, cached_upto = hashlib.sha256(), 0
for i, part in enumerate(parts): # longest previously-seen prefix is served from cache
prefix_hash.update(part.encode())
if prefix_hash.hexdigest() in cache:
cached_upto = i + 1
sizes = [approx_tokens(p) for p in parts]
read, new = sum(sizes[:cached_upto]), sum(sizes[cached_upto:])
total += (read * READ + new * WRITE) * PRICE_INPUT # new tokens are written to the cache
uncached += (read + new) * PRICE_INPUT # what the same request costs with no caching
hit_tokens += read
prefix_hash = hashlib.sha256()
for part in parts: # remember every prefix of this request
prefix_hash.update(part.encode()); cache.add(prefix_hash.hexdigest())
return total, uncached, hit_tokens
for layout in ["timestamp first", "stable first"]:
cost, uncached, hits = simulate(layout)
print(f"{layout:<16}: 20-turn chat input cost ${cost:.2f} with caching vs ${uncached:.2f} without "
f"| tokens served from cache {hits:,}")
- Timestamp first: a "current time" line at the very top changes on every request, so nothing ever matches. Worse than useless, it costs $2.56, more than the $2.05 with no caching at all, because every request pays the 1.25× write price.
- Stable first: tools, system prompt and history form a growing prefix that is reused every turn, and only the timestamp and the new turn are fresh. $0.34, about 6× cheaper than no caching, and the latency drops similarly.
The rules that follow.
- Put content in order of stability: tool definitions → system prompt → long reference documents → conversation history → the newest message and anything volatile (timestamps, request IDs).
- Freeze the stable parts: no timestamps or user names in the system prompt, a deterministic tool order, and JSON serialized with sorted keys.
- Changing tools, models or the system prompt mid-conversation breaks the cache. For instructions that arrive mid-conversation, append them as a new message instead of editing the top.
- Verify it works: the API reports cached tokens per request. If they stay at zero, something in your prefix is changing.
import anthropic
client = anthropic.Anthropic()
TOOLS = [...] # deterministic order, never rebuilt per request
SYSTEM = open("atlas_system_prompt.md").read() # frozen text: no dates, no user names
def ask(history, user_message, now):
response = client.messages.create(
model="claude-opus-5-5",
max_tokens=16000,
tools=TOOLS,
system=[{"type": "text", "text": SYSTEM, "cache_control": {"type": "ephemeral"}}],
messages=history + [
# breakpoint on the newest user turn: caches the whole conversation up to here
{"role": "user", "content": [{"type": "text", "text": user_message,
"cache_control": {"type": "ephemeral"}}]},
# volatile, operator-supplied state goes after the last breakpoint, as a mid-conversation
# system message (supported on current models), instead of editing the system prompt
{"role": "system", "content": f"Current time: {now}."},
],
)
u = response.usage
print(f"cache read {u.cache_read_input_tokens}, written {u.cache_creation_input_tokens}, uncached {u.input_tokens}")
return response
4. Tools, examples and other things that eat the window
The question. What else competes for context, and how do you keep it lean?
| Context item | Typical size | How to keep it lean |
|---|---|---|
| Tool definitions | 100–1,000 tokens each | Load only the tools relevant to the task (tool search / deferred loading); short, precise descriptions |
| Tool results | Up to tens of thousands | Return only needed fields; truncate with a "see more" handle; clear old results (context editing) |
| Few-shot examples | 100–500 each | Select dynamically (Prompt fundamentals §3); cache a fixed set |
| Retrieved documents | 200–1,000 per chunk | Better retrieval, re-ranking, dedup, a token budget (§1) |
| Conversation history | Grows without bound | Summaries, pinned facts, compaction (§2) |
| Files / images | 1,000+ each | Send only the pages or crops needed; reference by file ID instead of re-uploading |
Engineering reality. In agent workloads, tool results are usually the largest item. A single cat of a log file can be 50k tokens. Agents work best when tools return compact, structured results by default, and when the system clears or summarizes what has already been used (D9).
Quick check
An assistant's quality drops as conversations get long, even though the context window isn't full. What's a good mitigation?
Show the answer
C. Curate the context: summarize or drop stale turns, keep instructions and key facts in a stable prefix, and retrieve relevant history instead of including all of it — Irrelevant tokens dilute attention and cost money; curated, ordered context keeps the important information prominent.
Check yourself
- Your model supports 1M tokens. Give two reasons not to send your whole 400k-token knowledge base with every question.
- Prompt caching is enabled, but
cache_read_input_tokensis always 0. Name three likely causes. - Why can a badly ordered prompt cost more with caching enabled than with caching off?
- A sliding-window chatbot keeps forgetting the user's dietary restriction mentioned early on. Fix it without growing context without bound.
- Where in the context should the current timestamp go, and why?
Answers
- Cost and latency scale with input tokens (every request processes all 400k), and irrelevant material dilutes attention and adds distractors, which can reduce accuracy. Retrieve the relevant parts instead (D8).
- Something volatile early in the prefix (timestamp, request ID, user name in the system prompt); non-deterministic serialization (unsorted JSON, a different tool order each time); the tool set or model changing between requests; or a prefix shorter than the minimum cacheable length.
- Every request writes a new cache entry (about 1.25× the input price) but never reads one (0.1×), because the prefix never matches. You pay the write premium for nothing.
- Store it as a pinned fact (memory) that's included on every request, or retrieve user facts from a store. Old turns can still be summarized or dropped.
- At the end, after the stable prefix (for example in the latest message or as a mid-conversation system message). Anything after the point where it changes can't be cached, so placing it last keeps the rest cacheable.
Exercise
Write a ContextBuilder that every Atlas request goes through:
- A frozen prefix (system prompt + tool definitions) that is byte-for-byte identical on every request. Expose its hash so a test can assert it never changes.
- Then pinned facts, retrieved documents (best first), conversation history, and finally volatile content (the current time).
- A hard token budget: when over budget, drop the lowest-scoring documents first, then the oldest history turns, but always keep the last exchange. Raise an error if it still can't fit.
- Report what was dropped, so it can be logged.
Simulate 8 turns and assert, on every turn, that the budget holds and the prefix is unchanged.
Solution
import hashlib
approx_tokens = lambda text: max(1, len(text) // 4)
class ContextBuilder:
"""Stable, cacheable prefix first; per-request content after it; never over budget."""
def __init__(self, system, tools, budget):
self.prefix = [f"<system>{system}</system>", f"<tools>{tools}</tools>"] # frozen at startup
self.budget = budget
def prefix_hash(self):
return hashlib.sha256("".join(self.prefix).encode()).hexdigest()[:12]
def build(self, history, documents, pinned_facts, now):
"""history: list of (role, text), oldest first. documents: list of (score, text)."""
fixed = self.prefix + [f"<facts>{' '.join(pinned_facts)}</facts>"]
tail = [f"<now>{now}</now>"] # volatile content goes last
docs = sorted(documents, key=lambda d: -d[0])
hist = list(history)
dropped = {"documents": 0, "history turns": 0}
def assemble():
parts = fixed + [f"<doc>{t}</doc>" for _, t in docs] + [f"<{r}>{t}</{r}>" for r, t in hist] + tail
return parts, sum(approx_tokens(p) for p in parts)
parts, used = assemble()
while used > self.budget and (docs or len(hist) > 2):
if docs and (len(docs) > 1 or len(hist) <= 2): # weakest document goes first...
docs.pop(); dropped["documents"] += 1
else: # ...then the oldest turns (keep the last 2)
hist.pop(0); dropped["history turns"] += 1
parts, used = assemble()
if used > self.budget:
raise ValueError(f"cannot fit within {self.budget} tokens even after dropping optional content")
return parts, used, dropped
builder = ContextBuilder(system="You are Atlas. " * 40, tools="search_lessons(query) " * 30, budget=600)
history = []
for turn in range(8):
history += [("user", f"Question {turn} about KV caches " * 4), ("assistant", "A careful answer. " * 12)]
docs = [(0.9 - 0.1 * i, f"Lesson excerpt {i} about memory. " * 6) for i in range(4)]
parts, used, dropped = builder.build(history, docs, ["Student: GPU-infra interview next month."],
now=f"2026-11-11 10:{turn:02d}")
assert used <= builder.budget and parts[:2] == builder.prefix
if turn in (0, 3, 7):
print(f"turn {turn}: {used} tokens | dropped {dropped} | prefix {builder.prefix_hash()}")
The prefix hash is identical on every turn, so the provider can serve that part from cache. As the conversation grows, the builder sheds content in priority order: first the weaker documents (always keeping the best one), then old turns. That way the most valuable context survives, and the logs show exactly what each request was missing. In production, add a summary of dropped turns (§2), and count tokens with the API's token counter rather than the 4-characters estimate.
This unlocks:
- Prompt evaluation — measuring whether a context strategy actually improves answers.
- D8 Retrieval fundamentals — the retrieval step this lesson assumed: embeddings, chunking, vector search.
- D9 Agent memory — long-term memory and context management for long-running agents.
- D12 KV cache — the inference-side mechanism behind prompt caching.
Sources: Liu et al., Lost in the Middle: How Language Models Use Long Contexts (2023); Shi et al., Large Language Models Can Be Easily Distracted by Irrelevant Context (2023); Xu et al., Retrieval meets Long Context Large Language Models (2023); Packer et al., MemGPT: Towards LLMs as Operating Systems (2023); Anthropic, Prompt caching, Context windows, Compaction and Context editing documentation; Anthropic Engineering, Effective context engineering for AI agents (2025).