LangWBC: Language-directed Humanoid Whole-Body Control via End-to-end Learning

TL;DR

LangWBC tackles the challenge of translating natural language into agile, robust humanoid whole-body motions by introducing an end-to-end, language-conditioned control framework. It combines a two-stage training regime—a motion-tracking teacher trained via reinforcement learning on retargeted MoCap data and a CVAE-based student that binds language to actions in a unified latent space—enabling zero-shot sim-to-real transfer. The CVAE latent space supports diverse motion generation, smooth transitions, and interpolation to novel behaviors, while robustification techniques (PPO, domain randomization, symmetry) improve real-world reliability. The approach is demonstrated on real hardware with a broad set of motions, robustness to disturbances, and the ability to integrate with LLMs for complex task planning, highlighting its potential as a foundation model for humanoid control.

Abstract

General-purpose humanoid robots are expected to interact intuitively with humans, enabling seamless integration into daily life. Natural language provides the most accessible medium for this purpose. However, translating language into humanoid whole-body motion remains a significant challenge, primarily due to the gap between linguistic understanding and physical actions. In this work, we present an end-to-end, language-directed policy for real-world humanoid whole-body control. Our approach combines reinforcement learning with policy distillation, allowing a single neural network to interpret language commands and execute corresponding physical actions directly. To enhance motion diversity and compositionality, we incorporate a Conditional Variational Autoencoder (CVAE) structure. The resulting policy achieves agile and versatile whole-body behaviors conditioned on language inputs, with smooth transitions between various motions, enabling adaptation to linguistic variations and the emergence of novel motions. We validate the efficacy and generalizability of our method through extensive simulations and real-world experiments, demonstrating robust whole-body control. Please see our website at LangWBC.github.io for more information.

Figures (11)

• Figure 1:
• Figure 2: Robustness to External Disturbances. The humanoid robot demonstrates robust stability while executing a hand-waving motion under external perturbations. When subjected to kicks (top row) and pushes (bottom row), the robot maintains balance and continues the commanded motion, showcasing effective disturbance rejection capabilities without interrupting the primary task.
• Figure 3: Real World Demonstration. Conditioned on text commands, our framework is able to learn a diverse distribution of whole-body motions in action generation directly, and can be zero-shot deployed on real-world robots. More results are shown in the accompanying video.
• Figure 4: t-SNE Analysis of Latent Space. The plot shows 9 motions from 4 categories of motion, as shown in the legend. We see that similar motions (in the same color band) are closer than dissimilar ones. The axes suggest an interpretable structure: lateral symmetry (left/right motions mirrored across the y-axis) and vertical hierarchy (upper-body motions cluster at higher y-values, lower-body motions at lower y-values). We observe that all motions share a common region near the origin (0,0), likely representing a typical standing posture.
• Figure 5: Rollouts of Unseen Text Commands. Our method can generalize to unseen text commands with similar semantical meanings. Of the three, only one command "A person walks forward" (a) is included in the training dataset.