You already know: vision transformers and patch embeddings (Computer vision); embeddings, cosine similarity and contrastive training (D8 Embeddings); decoder-only transformers (D5); retrieval and document ingestion (D8).
Why this matters for Atlas. Two ideas power almost every modern multimodal system. Contrastive alignment (CLIP) trains an image encoder and a text encoder so that matching pairs land close together in one embedding space: that gives zero-shot classification and image–text search. Multimodal LLMs turn an image into a sequence of tokens that a language model reads alongside text: that gives open-ended questions about any image, including documents, charts and screenshots. This lesson builds contrastive learning from scratch, explains the architecture and token cost of VLMs, and covers document understanding (OCR vs reading pages directly) and how VLMs fail.
1. Contrastive alignment: CLIP from first principles
The question. How do you make an image and its caption have the same embedding, without labelled classes?
Start from nothing. Take a batch of N (image, caption) pairs from the web. Encode images with one network and captions with another, normalize, and compute the N × N matrix of cosine similarities. The diagonal holds matching pairs; everything off the diagonal is a mismatch. The contrastive (InfoNCE) loss is cross-entropy over each row (pick the right caption for this image) and each column (pick the right image for this caption), with a temperature that sharpens the softmax. Nothing in the loss names a class: the supervision is just "these two go together". Trained on hundreds of millions of pairs, the shared space captures concepts that transfer to new tasks.