Learning a Unified Latent Space for Cross-Embodiment Robot Control

TL;DR

Cross-embodiment control across humans and diverse humanoid platforms is addressed by a two-stage framework that first learns a segment-decomposed latent space and then trains a goal-conditioned policy in that shared space. The latent space uses segment-specific encoders/decoders with tailored similarity metrics, enabling alignment across morphologies; the control policy is a conditional variational autoencoder that predicts latent displacements $d_t = z_{t+1} - z_t$ conditioned on $z_t$ and an intention vector derived from end-effector velocity toward a user goal. New robots are added by learning only lightweight robot-specific embedding layers, keeping core networks fixed. Experiments show robust, embodiment-agnostic control across six humanoid platforms with sub-centimeter end-effector accuracy and real-time performance.

Abstract

We present a scalable framework for cross-embodiment humanoid robot control by learning a shared latent representation that unifies motion across humans and diverse humanoid platforms, including single-arm, dual-arm, and legged humanoid robots. Our method proceeds in two stages: first, we construct a decoupled latent space that captures localized motion patterns across different body parts using contrastive learning, enabling accurate and flexible motion retargeting even across robots with diverse morphologies. To enhance alignment between embodiments, we introduce tailored similarity metrics that combine joint rotation and end-effector positioning for critical segments, such as arms. Then, we train a goal-conditioned control policy directly within this latent space using only human data. Leveraging a conditional variational autoencoder, our policy learns to predict latent space displacements guided by intended goal directions. We show that the trained policy can be directly deployed on multiple robots without any adaptation. Furthermore, our method supports the efficient addition of new robots to the latent space by learning only a lightweight, robot-specific embedding layer. The learned latent policies can also be directly applied to the new robots. Experimental results demonstrate that our approach enables robust, scalable, and embodiment-agnostic robot control across a wide range of humanoid platforms.

Figures (12)

• Figure 1: Learning a Unified Latent Representation. Our architecture learns a shared latent space that unifies motion representations across diverse embodiments, including humans and various robots. To accurately model local motion patterns, we decouple the latent space into five subspaces corresponding to distinct body segments: left arm (LA), right arm (RA), trunk (TK), left leg (LL), and right leg (RL). The model comprises a human encoder ($E_h$), a cross-embodiment encoder ($E_X$), and a cross-embodiment decoder ($D_X$). To accommodate differences in pose dimensionality across robot platforms, each robot is assigned a learnable, robot-specific embedding layer $E_r$, which projects the raw pose representation into a shared cross-embodiment feature space. Conversely, $D_r$ is the inverse mapping back to the original pose space.
• Figure 2: Goal-conditioned Latent-Space Robot Control. Our c-VAE framework learns to model goal-directed motion dynamics in our shared latent space purely from human demonstrations. During training, the model conditions on the current latent pose $z_t$ and the average EE velocity toward a sampled goal $\overline{v}_{ee}$ to predict the latent displacement $d_t$. At inference time, $\overline{v}_{ee}$ is derived from the current robot pose and the user-specified goal position and time horizon. The decoder then generates the latent displacement $\hat{d}_t$ using $z_t$, $\overline{v}_{ee}$, and sampled latent noise. This autoregressive process iteratively updates the latent state, enabling smooth and goal-directed motion generation across robot embodiments.
• Figure 3: Comparison of visual resemblance. We retarget various dynamic human motions onto different robots (H1, JVRC, and TIAGo, from left to right), and compare different retargeting models. The result shows that both ImitationNet and our method with a decoupled latent space obtain high-quality visual resemblance. Our method trains a single model on all robots, while ImitationNet overfits each robot to a separate model.
• Figure 4: Cross-embodiment Motion Retargeting. We translate motions between any embodiments, and the results showcase the capability of our method, capturing the motion semantics across diverse embodiments.
• Figure 5: Latent-Space Multi-Robot Control. A single policy controls the latent space to enable each robot to reach arbitrary goal positions (blue) from various starting poses (green). Intermediate waypoints along the generated trajectories are shown in purple.