You already know: the decoder-only block with causal attention, GQA, RoPE and SwiGLU (Transformer architecture); masked vs autoregressive objectives (NLP foundations §5); cross-attention (Transformer architecture §2).
Why this matters for Atlas. Model names change every month; the design choices underneath don't. Each family makes a different trade between understanding vs generating, parameters vs compute per token, and context length vs memory. Knowing the trade-offs lets you pick the right model for search, chat or long documents — and predict what will be expensive to serve.
| Family | Attention pattern | Trained to | Best at | Examples |
|---|---|---|---|---|
| Encoder-only | bidirectional | fill in masked tokens | embeddings, classification, reranking | BERT, RoBERTa, many embedding models |
| Decoder-only | causal | predict the next token | generation, chat, code, in-context learning | GPT, LLaMA, Mistral, Claude-style LLMs |
| Encoder–decoder | bidirectional encoder + causal decoder with cross-attention | map input sequence → output sequence | translation, summarization, speech-to-text | T5, BART, Whisper |
| Mixture of experts | (any) + routed sparse MLPs | same as base | huge capacity at moderate compute per token | Mixtral, DeepSeek-V3, Switch Transformer |
| State-space / recurrent | no attention; linear-time recurrence | next token | very long sequences, constant memory per step | S4, Mamba, RWKV |
| Hybrid | mix of attention and SSM/local layers | next token | long context at lower cost | Jamba, models with sliding-window + global layers |
1. Encoder-only: reading in both directions
The question. Why are embedding and reranking models usually encoders rather than GPT-style decoders?
Start from nothing. An encoder lets every token attend to all tokens, left and right. Trained with masked language modelling, it builds rich representations of the whole input at once — ideal when you need to understand text, not continue it. To get one vector for a sentence (an embedding), pool the token vectors (mean, or a special [CLS] token) and fine-tune with a contrastive objective so similar sentences end up close (D8).