Augmented Reality for RObots (ARRO): Pointing Visuomotor Policies Towards Visual Robustness

TL;DR

ARRO tackles visuomotor robustness under visual domain shifts by transforming input frames into a task-focused augmented reality view using open-vocabulary segmentation and vision-language reasoning. It masks non-task regions and overlays them on a structured virtual background, enabling zero-shot applicability without camera calibration or retraining. Across real-world and simulated tasks, ARRO yields consistent performance gains for both task-specific policies and generalist models, improving resilience to background variations, distractors, and occlusions while supporting cross-embodiment transfer. This approach reduces data collection needs and offers a scalable preprocessing step compatible with a wide range of visuomotor policies.

Abstract

Visuomotor policies trained on human expert demonstrations have recently shown strong performance across a wide range of robotic manipulation tasks. However, these policies remain highly sensitive to domain shifts stemming from background or robot embodiment changes, which limits their generalization capabilities. In this paper, we present ARRO, a novel visual representation that leverages zero-shot open-vocabulary segmentation and object detection models to efficiently mask out task-irrelevant regions of the scene in real time without requiring additional training, modeling of the setup, or camera calibration. By filtering visual distractors and overlaying virtual guides during both training and inference, ARRO improves robustness to scene variations and reduces the need for additional data collection. We extensively evaluate ARRO with Diffusion Policy on a range of tabletop manipulation tasks in both simulation and real-world environments, and further demonstrate its compatibility and effectiveness with generalist robot policies, such as Octo and OpenVLA. Across all settings in our evaluation, ARRO yields consistent performance gains, allows for selective masking to choose between different objects, and shows robustness even to challenging segmentation conditions. Videos showcasing our results are available at: https://augmented-reality-for-robots.github.io/

Figures (9)

• Figure 1: Visualization of input formats (top) and performance comparison across four manipulation tasks (bottom). (a) shows the training scene, while (b) depicts an altered scene with visual domain shifts used during inference. (c) and (d) illustrate the corresponding inputs for the masked diffusion policy and ARRO, respectively.
• Figure 2: ARRO in a nutshell: Our pipeline segments the robot gripper and task-related using open-vocabulary vision models and overlays them onto a virtual background. We consistently use this process across training and inference to enhance robustness against visual domain shifts.
• Figure 3: Real-world experiment setups for the (a) pick-v1, (b) push-v1, (c) doll-v1 and (d) box-v1 tasks.
• Figure 4: Execution sequences using ARRO on the box-v1 and doll-v1 tasks. By overlaying the segmented task-relevant regions on a virtual background, ARRO neutralizes the effect of visual domain shifts.
• Figure 5: ARRO’s segmentation remains robust to transient occlusions caused by the robot arm or other objects. The doll is temporarily occluded by the octopus plush toy and the gripper in frames (5)–(7), but its segmentation accurately reappears once the occlusion clears, without manual correction.