From Language to Locomotion: Retargeting-free Humanoid Control via Motion Latent Guidance

TL;DR

RoboGhost presents a retargeting-free, latent-driven approach to language-guided humanoid locomotion that bypasses explicit motion decoding and morphology retargeting. It combines a continuous autoregressive motion generator with a causal transformer–diffusion framework, enabling a diffusion-based student policy that denoises executable actions directly from noise conditioned on language latents. A MoE-based teacher policy, trained with privileged simulation data, improves generalization, while a causal adaptive sampling strategy focuses learning on challenging, long-horizon segments. Across sim-to-real deployments on Unitree G1, RoboGhost achieves faster deployment, higher success rates, and reduced tracking error, and the framework extends naturally to other modalities beyond text, offering a universal approach for vision-language-action humanoid systems.

Abstract

Natural language offers a natural interface for humanoid robots, but existing language-guided humanoid locomotion pipelines remain cumbersome and untrustworthy. They typically decode human motion, retarget it to robot morphology, and then track it with a physics-based controller. However, this multi-stage process is prone to cumulative errors, introduces high latency, and yields weak coupling between semantics and control. These limitations call for a more direct pathway from language to action, one that eliminates fragile intermediate stages. Therefore, we present RoboGhost, a retargeting-free framework that directly conditions humanoid policies on language-grounded motion latents. By bypassing explicit motion decoding and retargeting, RoboGhost enables a diffusion-based policy to denoise executable actions directly from noise, preserving semantic intent and supporting fast, reactive control. A hybrid causal transformer-diffusion motion generator further ensures long-horizon consistency while maintaining stability and diversity, yielding rich latent representations for precise humanoid behavior. Extensive experiments demonstrate that RoboGhost substantially reduces deployment latency, improves success rates and tracking precision, and produces smooth, semantically aligned locomotion on real humanoids. Beyond text, the framework naturally extends to other modalities such as images, audio, and music, providing a universal foundation for vision-language-action humanoid systems.

Figures (8)

• Figure 1: RoboGhost is a retargeting-free latent driven policy for language-guided humanoid locomotion. By removing the dependency on motion retargeting, it thus allows robots to be controlled directly via open-ended language commands. The figure showcases (a) the previous pipeline with motion retargeting, (b) our proposed retargeting-free latent-driven pipeline, (c) quantitative comparisons of success rate and time cost between baseline and RoboGhost, (d) performing the backflip, and (e) dancing and leaping forward.
• Figure 2: Overview of RoboGhost. We propose a two-stage approach: a motion latent is first generated, then a MoE-based teacher policy is trained with RL and a diffusion-based student policy is trained to denoise actions conditioned on motion latent. This latent-driven scheme bypasses the need for motion retargeting.
• Figure 3: Qualitative results in the IsaacGym and MuJoCo.
• Figure 4: Qualitative results in the IsaacGym and MuJoCo.
• Figure 5: Qualitative results on the Unitree G1.