VFlash: Fast, Accurate Visual Planning via Flash Primitive Decoding
u_Unrolled
D9: AI Agents & Tool Use · Reasoning Failures — How Agents Go Wrong, and How to Catch It
Phase 5/Domain 9/Lesson 8

Reasoning Failures — How Agents Go Wrong, and How to Catch It

20 min read·Advanced· Unrolled Pro
Reasoning Failures — How Agents Go Wrong, and How to Catch It

You already know: preference training rewards what raters like, not only what's true (D6 Preference optimization); retrieved context can be irrelevant (D8 Advanced RAG); verifiers can be fooled and self-critique needs grounding (Reasoning & test-time compute, Agent architectures); traces record every step (Advanced agent engineering).

Why this matters for Atlas. Agents fail in a small number of recurring ways, and most aren't "the model isn't smart enough". They're systematic: agreeing with the user when the user is wrong, sticking to an early mistake, looping, getting distracted by irrelevant context, and "verifying" errors with the same reasoning that produced them. Each has a mechanism, a way to measure it, and a mitigation. Knowing the taxonomy turns "the agent is flaky" into a specific, fixable bug, and it's a favourite topic in agent-reliability interviews.


1. Sycophancy: agreeing with a wrong premise

The question. Why does Atlas back down when a student pushes back on a correct answer?

Start from nothing. Models trained on human preferences learn that agreeing tends to be rated well. The result is sycophancy: accepting false premises ("since 0.1 + 0.2 equals 0.3 exactly in floating point..."), changing a correct answer under pushback ("are you sure?"), or tailoring claims to the user's stated opinion. For a tutor this is the worst possible failure: it confirms misconceptions.

That's the preview. The full lesson, with its code and exercises, is part of Unrolled Pro.
Staff AI & Pro Tier Lesson

Unlock "Reasoning Failures — How Agents Go Wrong, and How to Catch It"

The foundations are free forever. Unrolled Pro unlocks the complete lesson, production architectures, and 55 battle-tested ML system design case studies.

Everything included with Pro:
All 234+ Deep-Dive Lessons across transformer internals, inference, RAG & agents
55 Staff-Level ML & System Design Case Studies with mathematical capacity estimates
AI Engineer Sheet & Exercises with worked solutions, self-checks, and code
100 Landmark Research Papers guided walkthroughs & progress tracking
14-Day Money-Back Guarantee — 100% full refund if you aren't satisfied
View Plans & Unlock All Lessons

From ₹167/mo (~$2/mo) · Cancel anytime · Instant activation

100% Risk-Free: 14-day no-questions-asked money-back guarantee
Build Atlas · Part 8 of 9: Let it actUse this lesson in the project: Give Atlas tools, memory and a loop, then put a gateway between the model and the tools so that a fooled model still can't do damage.