Sim2real Image Translation Enables Viewpoint-Robust Policies from Fixed-Camera Datasets

TL;DR

The paper tackles the brittleness of vision-based robot policies to camera viewpoint shifts by introducing MANGO, a sim2real image translation framework trained on unpaired simulation and fixed-camera real data. It deploys segmentation-conditioned InfoNCE (SegNCE), a modified PatchNCE loss, and a regularized discriminator to preserve unseen viewpoints during translation, enabling realistic diversification of simulated demonstrations. Empirically, MANGO yields improved FID and enhances downstream policy robustness to viewpoint changes, achieving up to $60\%$ success on shifted views and outperforming several baselines, including diffusion-based augmentation in computing efficiency. The approach provides a practical path to scalable, viewpoint-robust robot imitation learning without extensive real-world data collection or heavy computational resources, with potential for integration into larger pretrained models.

Abstract

Vision-based policies for robot manipulation have achieved significant recent success, but are still brittle to distribution shifts such as camera viewpoint variations. Robot demonstration data is scarce and often lacks appropriate variation in camera viewpoints. Simulation offers a way to collect robot demonstrations at scale with comprehensive coverage of different viewpoints, but presents a visual sim2real challenge. To bridge this gap, we propose MANGO -- an unpaired image translation method with a novel segmentation-conditioned InfoNCE loss, a highly-regularized discriminator design, and a modified PatchNCE loss. We find that these elements are crucial for maintaining viewpoint consistency during sim2real translation. When training MANGO, we only require a small amount of fixed-camera data from the real world, but show that our method can generate diverse unseen viewpoints by translating simulated observations. In this domain, MANGO outperforms all other image translation methods we tested. Imitation-learning policies trained on data augmented by MANGO are able to achieve success rates as high as 60\% on views that the non-augmented policy fails completely on.

Figures (5)

• Figure 1: (a) Standard image translation methods fail to generalize to new viewpoints when trained on fixed-viewpoint target domain dataset. (b) Our method, MANGO, enables realistic generation of unseen viewpoints, which are used to improve the robustness of downstream robot polices.
• Figure 2: MANGO is trained on unpaired real and sim images, specifically a real dataset obtained with a fixed camera and a simulation dataset with diverse camera viewpoints. To ensure the simulation viewpoint is preserved during translation, we employ a novel segmentation-based InfoNCE loss, a modified PatchNCE loss from CUT, and a random patch sampling and rotation process to regularize the discriminator $D$.
• Figure 3: Our real-world robot setup, including a Franka Emika Panda arm, a wrist camera, and an external camera that is only repositioned for evaluations.
• Figure 4: Sim2Sim training data with translations by MANGO for three of the tasks included in Table \ref{['tab:sim_results']}
• Figure 5: Left: Sample image observations for each task and data-augmentation method from Table \ref{['tab:real_results']}. Right: The three shifted viewpoints that comprise our "Shifted Cams" evaluations in Table \ref{['tab:real_results']}