You already know: images, detection and segmentation (Computer vision); video as a stream (Video intelligence); agent loops of observe → decide → act and guardrails for tools (D9).
Why this matters for Atlas. An agent that acts in the physical world adds three things text agents never face. Geometry: decisions need 3D positions, not just labels, so cameras must be calibrated and pixels lifted into space. Time: control runs at tens to hundreds of hertz, so a model that takes a second per decision can't be in the inner loop. Irreversibility: a dropped beaker can't be rolled back, so safety must be designed into the architecture. This lesson builds the camera model and depth back-projection, a feedback controller, the layered architecture of modern robot systems, vision-language-action models, and the sim-to-real gap.
1. From pixels to 3D: the pinhole camera
The question. A detector says "mug at pixel (412, 230)". Where is the mug, in metres?
Start from nothing. The pinhole camera model projects a 3D point (X, Y, Z) in the camera frame onto the image at u = f_x·X/Z + c_x, v = f_y·Y/Z + c_y, where f_x, f_y are focal lengths in pixels and (c_x, c_y) the principal point: the intrinsics, measured once by calibration (photographing a checkerboard). Projection loses depth: every point along a ray lands on the same pixel. To go back from pixel to 3D you need depth Z, from a depth camera (structured light, time-of-flight), stereo (two cameras: depth = focal × baseline / disparity) or a monocular depth model. Then X = (u − c_x)·Z/f_x, Y = (v − c_y)·Z/f_y. The extrinsics (camera pose relative to the robot) transform that point into the robot's coordinate frame.