VFlash: Fast, Accurate Visual Planning via Flash Primitive Decoding
u_Unrolled
D18: Multimodal Foundation Models · Spatial AI & Robotics — 3D Perception, Control Loops and Vision-Language-Action Models

Spatial AI & Robotics — 3D Perception, Control Loops and Vision-Language-Action Models

15 min read·Advanced· Unrolled Pro
Spatial AI & Robotics — 3D Perception, Control Loops and Vision-Language-Action Models

You already know: images, detection and segmentation (Computer vision); video as a stream (Video intelligence); agent loops of observe → decide → act and guardrails for tools (D9).

Why this matters for Atlas. An agent that acts in the physical world adds three things text agents never face. Geometry: decisions need 3D positions, not just labels, so cameras must be calibrated and pixels lifted into space. Time: control runs at tens to hundreds of hertz, so a model that takes a second per decision can't be in the inner loop. Irreversibility: a dropped beaker can't be rolled back, so safety must be designed into the architecture. This lesson builds the camera model and depth back-projection, a feedback controller, the layered architecture of modern robot systems, vision-language-action models, and the sim-to-real gap.


1. From pixels to 3D: the pinhole camera

The question. A detector says "mug at pixel (412, 230)". Where is the mug, in metres?

Start from nothing. The pinhole camera model projects a 3D point (X, Y, Z) in the camera frame onto the image at u = f_x·X/Z + c_x, v = f_y·Y/Z + c_y, where f_x, f_y are focal lengths in pixels and (c_x, c_y) the principal point: the intrinsics, measured once by calibration (photographing a checkerboard). Projection loses depth: every point along a ray lands on the same pixel. To go back from pixel to 3D you need depth Z, from a depth camera (structured light, time-of-flight), stereo (two cameras: depth = focal × baseline / disparity) or a monocular depth model. Then X = (u − c_x)·Z/f_x, Y = (v − c_y)·Z/f_y. The extrinsics (camera pose relative to the robot) transform that point into the robot's coordinate frame.

That's the preview. The full lesson, with its code and exercises, is part of Unrolled Pro.
Staff AI & Pro Tier Lesson

Unlock "Spatial AI & Robotics — 3D Perception, Control Loops and Vision-Language-Action Models"

The foundations are free forever. Unrolled Pro unlocks the complete lesson, production architectures, and 55 battle-tested ML system design case studies.

Everything included with Pro:
All 234+ Deep-Dive Lessons across transformer internals, inference, RAG & agents
55 Staff-Level ML & System Design Case Studies with mathematical capacity estimates
AI Engineer Sheet & Exercises with worked solutions, self-checks, and code
100 Landmark Research Papers guided walkthroughs & progress tracking
14-Day Money-Back Guarantee — 100% full refund if you aren't satisfied
View Plans & Unlock All Lessons

From ₹167/mo (~$2/mo) · Cancel anytime · Instant activation

100% Risk-Free: 14-day no-questions-asked money-back guarantee