Robot Learning from Any Images

TL;DR

RoLA addresses the data bottleneck in robot learning by turning any single image into a physically grounded, interactive robotic scene. It combines Real-to-Sim scene recovery, scalable Simulation-based data generation, and Sim-to-Real photorealistic data synthesis via visual blending to produce large-scale visuomotor demonstrations without specialized hardware. The approach enables single-image robot learning, real-world deployment across manipulators and a humanoid, and training of vision-language-action models, with demonstrated benefits from Internet image priors. This work highlights the potential of leveraging in-the-wild visual data to enable scalable, diverse, and practical robot learning.

Abstract

We introduce RoLA, a framework that transforms any in-the-wild image into an interactive, physics-enabled robotic environment. Unlike previous methods, RoLA operates directly on a single image without requiring additional hardware or digital assets. Our framework democratizes robotic data generation by producing massive visuomotor robotic demonstrations within minutes from a wide range of image sources, including camera captures, robotic datasets, and Internet images. At its core, our approach combines a novel method for single-view physical scene recovery with an efficient visual blending strategy for photorealistic data collection. We demonstrate RoLA's versatility across applications like scalable robotic data generation and augmentation, robot learning from Internet images, and single-image real-to-sim-to-real systems for manipulators and humanoids. Video results are available at https://sihengz02.github.io/RoLA .

Figures (16)

• Figure 1: RoLA transforms a single in-the-wild image into an interactive, physics-enabled robotic environment. Given a single input image (top-left), RoLA recovers the physical scene for robot learning (top-right), enables large-scale robotic data generation (bottom-right), and supports deployment of learned policies on real robots (bottom-left).
• Figure 2: An overview of the RoLA framework. Step 1: Recover the physical scene from a single image. Step 2: Generate large-scale photorealistic robotic demonstrations via visual blending. Step 3: Train and deploy policies across tasks and embodiments using the collected data.
• Figure 3: (a) Real-world deployment of policies trained with RoLA-generated data. (b) RoLA enables efficient real-to-sim-to-real transfer for humanoid robots.
• Figure 4: Real-world execution of a VLA model trained with RoLA-generated data.
• Figure 5: Learning a vision-based apple grasping prior from Internet apple images.