CEI: A Unified Interface for Cross-Embodiment Visuomotor Policy Learning in 3D Space

TL;DR

CEI addresses embodiment biases in robotic manipulation by introducing functional similarity and the Directional Chamfer Distance to align source demonstrations to target morphologies. Through gradient-based trajectory alignment and observation/action synthesis, CEI enables offline cross-embodiment transfer across 16 simulated embodiments and bidirectional real-world transfers with an average transfer ratio around 82.4%, using DP3 policies trained on synthesized data. The method supports spatial generalization and multimodal motion generation, demonstrating practical impact for scalable learning of visuomotor policies across diverse end-effectors, including dexterous hands. While promising, future work could integrate tactile sensing and RGB-based inputs to further robustify cross-embodiment learning in physically intricate tasks.

Abstract

Robotic foundation models trained on large-scale manipulation datasets have shown promise in learning generalist policies, but they often overfit to specific viewpoints, robot arms, and especially parallel-jaw grippers due to dataset biases. To address this limitation, we propose Cross-Embodiment Interface (\CEI), a framework for cross-embodiment learning that enables the transfer of demonstrations across different robot arm and end-effector morphologies. \CEI introduces the concept of \textit{functional similarity}, which is quantified using Directional Chamfer Distance. Then it aligns robot trajectories through gradient-based optimization, followed by synthesizing observations and actions for unseen robot arms and end-effectors. In experiments, \CEI transfers data and policies from a Franka Panda robot to \textbf{16} different embodiments across \textbf{3} tasks in simulation, and supports bidirectional transfer between a UR5+AG95 gripper robot and a UR5+Xhand robot across \textbf{6} real-world tasks, achieving an average transfer ratio of 82.4\%. Finally, we demonstrate that \CEI can also be extended with spatial generalization and multimodal motion generation capabilities using our proposed techniques. Project website: https://cross-embodiment-interface.github.io/

Figures (7)

• Figure 1: Cross-embodiment interface. CEI enables cross-embodiment transfer between different robots by synthesizing demonstrations from a source embodiment to a target embodiment. We transfer data and policies from a Franka Panda robot to 16 target embodiments across 3 tasks in simulation, and demonstrate bidirectional transfer between a UR5+AG95 gripper and a UR5+Xhand setup across 6 real-world tasks. We also showcase CEI's compatibility with spatial generalization and multimodal motion generation.
• Figure 2: Overview of the pipeline. Given a source dataset, a source embodiment, and a target embodiment, we first define functional representations as sets of points with associated directions on both embodiments. We then compute functional similarity using the negative Directional Chamfer Distance between these representations. Trajectory alignment is performed by sequentially optimizing the functional similarity for each trajectory slice. Finally, we synthesize target actions with next joint positions and generate target observations by augmenting source point clouds with points sampled from the target embodiment. The viridis colormap is used to illustrate the temporal progression of the trajectory of functional representations.
• Figure 3: Tasks and embodiments for simulation evaluation. We investigate 3 tasks and 16 embodiments, which are combinations of 4 robot arms (UR5e, IIWA, Kinova3 and Franka Panda) and 4 end-effectors (FourierRighthand, InspireRightHand, FourierLefthand and RobotiqThreeFinger gripper).
• Figure 4: Left: Setup and associated objects in real-world experiments. Right: Real-world tasks. We evaluate transfer from the AG95 gripper to the Xhand on PushCube, OpenDrawer, and PlaceBird, and from the Xhand to the AG95 gripper on PickCup, PackageBread, and InsertFlower.
• Figure 5: Qualitative evaluation. Left: Transfer from AG95 to Xhand in PushCube, OpenDrawer, and PlaceBird. Right: Transfer from Xhand to AG95 in PickCup, PackageBread, and InsertFlower. Manipulations of source policies are shown on the top rows, transferred ones on the bottom rows.