FRoM-W1: Towards General Humanoid Whole-Body Control with Language Instructions

TL;DR

FRoM-W1 addresses the challenge of language-guided, general-purpose humanoid motion by decoupling language understanding from physical execution in a two-stage pipeline: H-GPT translates natural language into coherent human whole-body motions using Chain-of-Thought reasoning and a VQ-VAE-based motion tokenizer, and H-ACT retargets these motions to robot morphologies and learns stable policies via RL in simulation before deployment on real robots. The framework introduces a 9B H-GPT model trained on enriched data (HumanML3D-X, Motion-X) with CoT to improve instruction understanding, and a two-stage RL-based controller (RPT and RFT) within a modular RoboJudo deployment that supports sim-to-real transfer and multi-policy compatibility. Key contributions include (i) functional language-to-motion generation with hand-inclusive SMPL-X representations, (ii) a robust human-to-humanoid retargeting pipeline, (iii) a teacher-student RL paradigm for sim-to-real whole-body control, and (iv) an open-source framework enabling plug-and-play deployment across diverse robots and controllers. The results show significant improvements in motion realism (FID), instruction generalization (CoT), and motion-tracking accuracy and task success rates in simulation and real-world tests on Unitree H1/G1, validating the practicality of language-guided humanoid control at scale.

Abstract

Humanoid robots are capable of performing various actions such as greeting, dancing and even backflipping. However, these motions are often hard-coded or specifically trained, which limits their versatility. In this work, we present FRoM-W1, an open-source framework designed to achieve general humanoid whole-body motion control using natural language. To universally understand natural language and generate corresponding motions, as well as enable various humanoid robots to stably execute these motions in the physical world under gravity, FRoM-W1 operates in two stages: (a) H-GPT: utilizing massive human data, a large-scale language-driven human whole-body motion generation model is trained to generate diverse natural behaviors. We further leverage the Chain-of-Thought technique to improve the model's generalization in instruction understanding. (b) H-ACT: After retargeting generated human whole-body motions into robot-specific actions, a motion controller that is pretrained and further fine-tuned through reinforcement learning in physical simulation enables humanoid robots to accurately and stably perform corresponding actions. It is then deployed on real robots via a modular simulation-to-reality module. We extensively evaluate FRoM-W1 on Unitree H1 and G1 robots. Results demonstrate superior performance on the HumanML3D-X benchmark for human whole-body motion generation, and our introduced reinforcement learning fine-tuning consistently improves both motion tracking accuracy and task success rates of these humanoid robots. We open-source the entire FRoM-W1 framework and hope it will advance the development of humanoid intelligence.

Figures (16)

• Figure 1: (a) We introduce FRoM-W1, an open-source framework that leverages Chain-of-Thought (CoT) reasoning to achieve general language-guided hand-inclusive humanoid whole-body control. (b) Based on a 9B instruction to human-motion model H-GPT, FRoM-W1 can generate diverse humanoid whole-body motions. (c) Through its H-ACT module which has been pre-trained and fine-tuned through reinforcement learning (RL), FRoM-W1 grounds generated human whole-body motions to various humanoid robots such as the Unitree H1, G1, and Fourier GR1T1, and enables them to execute corresponding motions accurately and stably in the physical world.
• Figure 2: The inference pipeline of FRoM-W1. (a) H-GPT first translates language instructions into fine-grained action descriptions through CoT, then generates corresponding whole-body human motion sequences. (b) After obtaining the human SMPL-X motions through inverse kinematics (IK), H-ACT retargets the generated human motion sequences into robot-specific motions. Then, through a reinforcement learning pre-trained (RPT) and fine-tuned (RFT) whole-body motion controller, H-ACT enables the humanoid robot to perform the corresponding actions in the physical world.
• Figure 3: Overview of the H-GPT. During training phase, data triplets <$\mathcal{I}$, $\mathcal{COT}$,$\mathcal{M}_h$> are tokenized and fed into the motion generator, where the motion sequences are encoded and discretized by the encoder $\mathcal{E}$ and the quantizer $Q$ of the whole-body motion tokenizer. During inference phase, the motion generator produces CoT and motion tokens given a specific instruction. These motion tokens are then decoded to a motion sequence by the decoder $\mathcal{D}$ of the tokenizer.
• Figure 4: Overview of the human-to-humanoid motion retargeting pipeline. (a) We first convert the 3D coordinates of the full-body keypoints into the SMPL-X representation expressed in axis-angle format. (b) For body retargeting, we follow PHC to align global body poses, and additionally incorporate a rotation loss to compute wrist joint orientations. (c) For hand retargeting, we formulate an optimization objective that aligns fingertip positions and solve it to obtain the final hand poses.
• Figure 5: Overview of the H-ACT. During the pre-training and fine-tuning phase, retargeted motion datasets are used in simulation to train a privileged teacher tracking policy. Through DAgger distillation, a sim-to-real student policy is trained with simplified motion goals; During the deployment phase, the generated humanoid motion is adapted to motion goals for different policy variants. A decoupled design with a unified interface enables seamless sim-to-sim transfer and sim-to-real deployment.