Unveiling the Impact of Data and Model Scaling on High-Level Control for Humanoid Robots

TL;DR

This work tackles data and model scaling for humanoid robot control by introducing Humanoid-Union, a large-scale, automatically generated dataset of executable robot motions with semantic annotations derived from human videos, and SCHUR, a two-stage framework that tokenizes robot motion with Finite Scalar Quantization and generates motion autoregressively conditioned on text prefixes. The approach improves tokenization quality, scales with larger codebooks and model sizes, and achieves better MPJPE and FID metrics, while directly validating viability on real-world humanoid robots via a universal whole-body tracker. By aligning robot motion with text as an intermediate modality, SCHUR enables high-level, scalable control and diverse action generation, demonstrated through extensive ablations and real-world deployment. The work highlights the practical benefits of large-scale, tracker-filtered robot motion data for learning-based control and points to future work in real-time, text-only control without trackers.

Abstract

Data scaling has long remained a critical bottleneck in robot learning. For humanoid robots, human videos and motion data are abundant and widely available, offering a free and large-scale data source. Besides, the semantics related to the motions enable modality alignment and high-level robot control learning. However, how to effectively mine raw video, extract robot-learnable representations, and leverage them for scalable learning remains an open problem. To address this, we introduce Humanoid-Union, a large-scale dataset generated through an autonomous pipeline, comprising over 260 hours of diverse, high-quality humanoid robot motion data with semantic annotations derived from human motion videos. The dataset can be further expanded via the same pipeline. Building on this data resource, we propose SCHUR, a scalable learning framework designed to explore the impact of large-scale data on high-level control in humanoid robots. Experimental results demonstrate that SCHUR achieves high robot motion generation quality and strong text-motion alignment under data and model scaling, with 37\% reconstruction improvement under MPJPE and 25\% alignment improvement under FID comparing with previous methods. Its effectiveness is further validated through deployment in real-world humanoid robot.

Figures (7)

• Figure 1: We present the large-scale, high-quality robot motion dataset Humanoid-Union to achieve data scaling. Additionally, we propose a text-based robot motion control method, SCHUR, which enables effective scalable learning. By scaling both data and models, our approach can generate complex, diverse and high-quality whole-body robot motions from text, which are executed by a whole-body tracker and successfully deployed on real-world robots.
• Figure 2: The data pipeline of Humanoid-Union. Pose data is extracted from videos by pose estimation, resulting in the SMPL representation. Corresponding descriptions are generated using vision-language model. The SMPL data is then retargeted to obtain the corresponding robot motion data, which is subsequently filtered and post-processed using a trained universal whole-body tracker.
• Figure 3: Comparison between the commonly used SMPL skeleton and our manually bound keypoints reveals that the virtual keypoints we defined exhibit a topology that is closer to SMPL. This alignment allows for a more accurate representation, maintaining a style that is more consistent with human motion.
• Figure 4: The framework of SCHUR consists of two stages. In the first stage, robot motion is tokenized using an effective representation, with FSQ employed for quantization. Here, we present an example using L=3 to illustrate FSQ. The second stage utilizes text tokens as prefix and applies prefix-bidirectional attention to generate motion tokens in an autoregressive manner.
• Figure 5: Comparison of SCHUR's tokenization stage with conventional VQ-VAE under different codebook sizes, using metrics such as MPJPE, MPKPE, and L1 loss. With the introduction of FSQ, SCHUR significantly outperforms conventional VQ-VAE in terms of reconstruction quality. Furthermore, it demonstrates strong scaling properties, with performance continuously improving as the codebook size increases. In contrast, VQ-VAE experiences instability during training with larger codebook sizes and eventually suffers from collapse.