RoboSwap: A GAN-driven Video Diffusion Framework For Unsupervised Robot Arm Swapping

TL;DR

RoboSwap tackles the challenge of swapping a robotic arm in a video when paired cross-embodiment data are unavailable. It fuses a GAN-based Stage 1 arm translation with a diffusion-based Stage 2 inpainting to produce realistic, motion-consistent swapped-arm videos, operated over unpaired datasets from diverse environments. The two stages are trained separately and then integrated at inference, using background-foreground decoupling, latent diffusion conditioned on reference video and textual prompts, and LoRA-tuned diffusion models. Across three robotic-arm benchmarks, RoboSwap outperforms state-of-the-art editing models in both structural coherence and motion realism, illustrating the approach’s potential to expand cross-embodiment data generation for robotic learning.

Abstract

Recent advancements in generative models have revolutionized video synthesis and editing. However, the scarcity of diverse, high-quality datasets continues to hinder video-conditioned robotic learning, limiting cross-platform generalization. In this work, we address the challenge of swapping a robotic arm in one video with another: a key step for crossembodiment learning. Unlike previous methods that depend on paired video demonstrations in the same environmental settings, our proposed framework, RoboSwap, operates on unpaired data from diverse environments, alleviating the data collection needs. RoboSwap introduces a novel video editing pipeline integrating both GANs and diffusion models, combining their isolated advantages. Specifically, we segment robotic arms from their backgrounds and train an unpaired GAN model to translate one robotic arm to another. The translated arm is blended with the original video background and refined with a diffusion model to enhance coherence, motion realism and object interaction. The GAN and diffusion stages are trained independently. Our experiments demonstrate that RoboSwap outperforms state-of-the-art video and image editing models on three benchmarks in terms of both structural coherence and motion consistency, thereby offering a robust solution for generating reliable, cross-embodiment data in robotic learning.

Figures (6)

• Figure 1: We propose RoboSwap, a GAN-driven Video Diffusion Approach to swap the robot arm in a video with another one. Contrary to previous works humantransmirage which swap the robot arms in the same environment, our approach enables the replacement of robotic arm across different domains, without requiring paired datasets. The generated videos conform to the motion dynamics of the reference robotic arm.
• Figure 2: RoboSwap has two stages, each trained independently. (A) First, we train a GAN cyclegan on two unpaired datasets of robotic arms, segmented from their backgrounds, to eliminate background interference and focus on pose translation between the robots only. (B) Second, A video diffusion model is then trained to refine the blended inputs of background and translated robotic arms. To mitigate GAN artifacts, distortion augmentations are applied to the arms during training. The diffusion model is trained on refining the videos in a self-supervised way on both datasets A and B. (C) During inference, the robotic arm is segmented, transformed via GAN, composited onto the background, and refined by the video diffusion model for enhanced realism.
• Figure 3: COMPARISON AGAINST DIFFERENT IMAGE EDITING APPROACHES.
• Figure 4: Comparison against different Video Editing Approaches for Robot Swap on the Kuka Robot → Google Robot task.
• Figure 5: Comparison against different Video Editing Approaches for Robot Swap on the Google Robot → UR5 Robot task.