VFlash: Fast, Accurate Visual Planning via Flash Primitive Decoding
u_Unrolled
D5: LLM Architecture · LLM Architecture Families — Trade-offs, Not Model Names
Phase 4/Domain 5/Lesson 4

LLM Architecture Families — Trade-offs, Not Model Names

15 min read·Advanced· Unrolled Pro
LLM Architecture Families — Trade-offs, Not Model Names

You already know: the decoder-only block with causal attention, GQA, RoPE and SwiGLU (Transformer architecture); masked vs autoregressive objectives (NLP foundations §5); cross-attention (Transformer architecture §2).

Why this matters for Atlas. Model names change every month; the design choices underneath don't. Each family makes a different trade between understanding vs generating, parameters vs compute per token, and context length vs memory. Knowing the trade-offs lets you pick the right model for search, chat or long documents — and predict what will be expensive to serve.

Family Attention pattern Trained to Best at Examples
Encoder-only bidirectional fill in masked tokens embeddings, classification, reranking BERT, RoBERTa, many embedding models
Decoder-only causal predict the next token generation, chat, code, in-context learning GPT, LLaMA, Mistral, Claude-style LLMs
Encoder–decoder bidirectional encoder + causal decoder with cross-attention map input sequence → output sequence translation, summarization, speech-to-text T5, BART, Whisper
Mixture of experts (any) + routed sparse MLPs same as base huge capacity at moderate compute per token Mixtral, DeepSeek-V3, Switch Transformer
State-space / recurrent no attention; linear-time recurrence next token very long sequences, constant memory per step S4, Mamba, RWKV
Hybrid mix of attention and SSM/local layers next token long context at lower cost Jamba, models with sliding-window + global layers

1. Encoder-only: reading in both directions

The question. Why are embedding and reranking models usually encoders rather than GPT-style decoders?

Start from nothing. An encoder lets every token attend to all tokens, left and right. Trained with masked language modelling, it builds rich representations of the whole input at once — ideal when you need to understand text, not continue it. To get one vector for a sentence (an embedding), pool the token vectors (mean, or a special [CLS] token) and fine-tune with a contrastive objective so similar sentences end up close (D8).

That's the preview. The full lesson, with its code and exercises, is part of Unrolled Pro.
Staff AI & Pro Tier Lesson

Unlock "LLM Architecture Families — Trade-offs, Not Model Names"

The foundations are free forever. Unrolled Pro unlocks the complete lesson, production architectures, and 55 battle-tested ML system design case studies.

Everything included with Pro:
All 234+ Deep-Dive Lessons across transformer internals, inference, RAG & agents
55 Staff-Level ML & System Design Case Studies with mathematical capacity estimates
AI Engineer Sheet & Exercises with worked solutions, self-checks, and code
100 Landmark Research Papers guided walkthroughs & progress tracking
14-Day Money-Back Guarantee — 100% full refund if you aren't satisfied
View Plans & Unlock All Lessons

From ₹167/mo (~$2/mo) · Cancel anytime · Instant activation

100% Risk-Free: 14-day no-questions-asked money-back guarantee
Build Atlas · Part 4 of 9: From text to GPTUse this lesson in the project: Train a byte-level BPE tokenizer and a 940K-parameter GPT on the lessons, on a laptop CPU, and read its loss curve honestly.