Mirage: Cross-Embodiment Zero-Shot Policy Transfer with Cross-Painting

TL;DR

Mirage enables zero-shot policy transfer across robot embodiments by decoupling vision and control: cross-painting masks the target robot and renders the source robot in its place, while a forward-dynamics model coupled with a blocking controller adapts source actions to the target. The approach is validated across nine manipulation tasks in simulation and real hardware, showing Mirage consistently outperforms a state-of-the-art generalist model and enabling robust transfer even with gripper and robot changes. Key contributions include a systematic simulation study, the cross-painting transfer strategy, and extensive real-world demonstrations, highlighting a practical path to reuse policies across diverse robotic platforms. The work suggests that leveraging robots’ URDFs and aligned end-effector action spaces can substantially reduce data collection and training needs for multi-robot manipulation.

Abstract

The ability to reuse collected data and transfer trained policies between robots could alleviate the burden of additional data collection and training. While existing approaches such as pretraining plus finetuning and co-training show promise, they do not generalize to robots unseen in training. Focusing on common robot arms with similar workspaces and 2-jaw grippers, we investigate the feasibility of zero-shot transfer. Through simulation studies on 8 manipulation tasks, we find that state-based Cartesian control policies can successfully zero-shot transfer to a target robot after accounting for forward dynamics. To address robot visual disparities for vision-based policies, we introduce Mirage, which uses "cross-painting"--masking out the unseen target robot and inpainting the seen source robot--during execution in real time so that it appears to the policy as if the trained source robot were performing the task. Mirage applies to both first-person and third-person camera views and policies that take in both states and images as inputs or only images as inputs. Despite its simplicity, our extensive simulation and physical experiments provide strong evidence that Mirage can successfully zero-shot transfer between different robot arms and grippers with only minimal performance degradation on a variety of manipulation tasks such as picking, stacking, and assembly, significantly outperforming a generalist policy. Project website: https://robot-mirage.github.io/

Figures (5)

• Figure 1: Simulation Tasks and Robots. The simulation evaluation utilizes the Robosuite simulator with Lift, Stack, Can Pick-and-Place, Two Piece Assembly, and Square Peg Insertion tasks. Additionally, to study other policy classes, we evaluate policy transfers in ORBIT with a block lifting task, and in RLBench with 2 tasks: Lifting a lid, and Pushing a button to turn on a lamp. For all policies, the source robot is the Franka robot as shown in the top row, while the target robots for each of the tasks are shown in the bottom row.
• Figure 2: Illustration of Mirage's pipeline. We reproject the camera from the target frame to the source frame if there is a non-negligible camera angle change and then apply cross-painting: (1) use the segmentation mask provided by Gazebo to mask out the target robot, (2) apply the fast marching telea2004image algorithm to fill in the missing pixels, and (3) overlay Gazebo's rendering of the source robot URDF onto the image. The resulting image is fed into the source robot's policy to obtain the action, which is executed after a coordinate frame transform.
• Figure 3: Trajectory Rollouts of Simulated (Left) and Real (Right) Tasks. For each task, the top row shows the actual observations of the target robot during the trajectory rollout, and the bottom row shows the cross-painted images generated by Mirage that are passed to the source robot policy $\pi_\mathcal{S}$ to obtain the desired target robot pose $p_{r, t+1}^\mathcal{T}$.
• Figure 4: Mirage applied to first-person wrist camera images and third-person front camera images. For each view, the top row shows the actual observations of the target robot during the rollout, and the bottom row shows the cross-painted images generated by Mirage.
• Figure 5: (a) An example of camera calibration error resulting in failure to mask all of the target robot out; (b) An example of the artifacts introduced due to large changes in camera angles.