Vision-Language-Action Robot Models
Vision-language-action models (RT-2 2023, pi0 2024, Gemini Robotics 2025) turn camera images and a spoken instruction directly into robot motion, using knowledge from web-scale language models.
Open in the interactive tree →Google's RT-2 (July 2023) wrote robot actions as text tokens and was trained on web data and robot trajectories together, so it followed commands that never appeared in its robot data. Physical Intelligence's pi0 (October 2024) joined a flow-matching action generator to a pretrained vision-language model and folded laundry, cleared tables and assembled boxes on several robot types. Gemini Robotics (March 2025) learns new short tasks from as few as 100 demonstrations and adapts to new robot bodies.
As of now
Physical Intelligence's pi0.7 (arXiv, 16 April 2026) is steered by prompts that can include subgoal images and performance metadata, and it learns from demonstrations, failed autonomous attempts and non-robot data. Its authors report that it folds laundry on a robot platform that never saw that task and runs an espresso machine out of the box. All reported results are the developers' own lab demonstrations on a few robot platforms.
Where today's AI could help
Medium AI leverage. The models are AI themselves and improve with language-model methods, but every gain needs real robot data and slow, risky testing in the physical world.
- Vision-language models supply object and scene knowledge learned from web data
- Language models break instructions into steps and write task plans
- Failed autonomous attempts and non-robot video are reused as extra training data
Shown so far
- RT-2 (July 2023) followed commands absent from its robot training data across 6,000 evaluation trials. source
- Gemini Robotics (March 2025) learns new short-horizon tasks from as few as 100 demonstrations. source
- pi0.7 (April 2026) learns from demonstrations, failed autonomous runs and non-robot data. source
Prerequisites
- Industrial Robot (Unimate)1961Robot arms and humanoids are the bodies these models control.
- Large Language Models2022Vision-language models supply the web knowledge and instruction following.
Unlocks
- Humanoid Robots2026Humanoid control stacks use vision-language-action models to follow spoken instructions.
- Everyday Robot DexterityopenGeneralist robot models are the current attempt at flexible grasping and folding.