RoVi-Aug: Robot and Viewpoint Augmentation for Cross-Embodiment Robot Learning

TL;DR

RoVi-Aug addresses the data scarcity and embodiment mismatch in robot learning by introducing diffusion-based robot and viewpoint augmentation. The method combines Ro-Aug (robot-to-robot image synthesis with segmentation and inpainting) and Vi-Aug (3D-aware viewpoint synthesis) to create rich, cross-embodiment training data, enabling zero-shot transfer and improved fine-tuning. Empirical results on Franka and UR5 across five tasks demonstrate comparable zero-shot performance to test-time adaptation baselines and show substantial gains when co-training on augmented data for multi-robot, multi-task policies. The work highlights practical gains in data efficiency and transferability, while also outlining limitations due to artifact generation and the need for broader gripper and background variability.

Abstract

Scaling up robot learning requires large and diverse datasets, and how to efficiently reuse collected data and transfer policies to new embodiments remains an open question. Emerging research such as the Open-X Embodiment (OXE) project has shown promise in leveraging skills by combining datasets including different robots. However, imbalances in the distribution of robot types and camera angles in many datasets make policies prone to overfit. To mitigate this issue, we propose RoVi-Aug, which leverages state-of-the-art image-to-image generative models to augment robot data by synthesizing demonstrations with different robots and camera views. Through extensive physical experiments, we show that, by training on robot- and viewpoint-augmented data, RoVi-Aug can zero-shot deploy on an unseen robot with significantly different camera angles. Compared to test-time adaptation algorithms such as Mirage, RoVi-Aug requires no extra processing at test time, does not assume known camera angles, and allows policy fine-tuning. Moreover, by co-training on both the original and augmented robot datasets, RoVi-Aug can learn multi-robot and multi-task policies, enabling more efficient transfer between robots and skills and improving success rates by up to 30%. Project website: https://rovi-aug.github.io.

Figures (8)

• Figure 1: Given robot images, RoVi-Aug uses state-of-the-art diffusion models to augment the data and generate synthetic images with different robots and viewpoints. Policy trained on the augmented dataset can be deployed on the target robots zero-shot or further finetuned, exhibiting robustness to camera pose changes.
• Figure 2: Overview of the RoVi-Aug pipeline. Given an input robot image, we first segment the robot out using a finetuned SAM kirillov2023segany model, then use a ControlNet zhang2023adding to transform the robot into another robot. After pasting the synthetic robot back into the background, we use ZeroNVS sargent2023zeronvs to generate novel views.
• Figure 3: Tasks used for evaluation. For each task, on the left is an example training view and robot, and on the right is the different test-time embodiment.
• Figure 4: Evaluated camera views. For static third-person cameras, we perturb the initial training view by 10 cm translation, 20$^{\circ}$ rotation and 25 cm translation, 35$^{\circ}$ rotation. Even when the camera is moving dynamically, RoVi-Aug is able to successfully sweep the cloth.
• Figure 5: Example of paired images for training the R2R model. We use Robosuite robosuite2020 to generate pairs of Jaco, Franka, Sawyer, and UR5 at the same pose.