Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Robust and Generalized Humanoid Motion Tracking

Yubiao Ma, Han Yu, Jiayin Xie, Changtai Lv, Qiang Luo, Chi Zhang, Yunpeng Yin, Boyang Xing, Xuemei Ren, Dongdong Zheng

TL;DR

A dynamics-conditioned command aggregation framework that uses a causal temporal encoder to summarize recent proprioception and a multi-head cross-attention command encoder to selectively aggregate a context window based on the current dynamics is proposed.

Abstract

Learning a general humanoid whole-body controller is challenging because practical reference motions can exhibit noise and inconsistencies after being transferred to the robot domain, and local defects may be amplified by closed-loop execution, causing drift or failure in highly dynamic and contact-rich behaviors. We propose a dynamics-conditioned command aggregation framework that uses a causal temporal encoder to summarize recent proprioception and a multi-head cross-attention command encoder to selectively aggregate a context window based on the current dynamics. We further integrate a fall recovery curriculum with random unstable initialization and an annealed upward assistance force to improve robustness and disturbance rejection. The resulting policy requires only about 3.5 hours of motion data and supports single-stage end-to-end training without distillation. The proposed method is evaluated under diverse reference inputs and challenging motion regimes, demonstrating zero-shot transfer to unseen motions as well as robust sim-to-real transfer on a physical humanoid robot.

Robust and Generalized Humanoid Motion Tracking

TL;DR

A dynamics-conditioned command aggregation framework that uses a causal temporal encoder to summarize recent proprioception and a multi-head cross-attention command encoder to selectively aggregate a context window based on the current dynamics is proposed.

Abstract

Learning a general humanoid whole-body controller is challenging because practical reference motions can exhibit noise and inconsistencies after being transferred to the robot domain, and local defects may be amplified by closed-loop execution, causing drift or failure in highly dynamic and contact-rich behaviors. We propose a dynamics-conditioned command aggregation framework that uses a causal temporal encoder to summarize recent proprioception and a multi-head cross-attention command encoder to selectively aggregate a context window based on the current dynamics. We further integrate a fall recovery curriculum with random unstable initialization and an annealed upward assistance force to improve robustness and disturbance rejection. The resulting policy requires only about 3.5 hours of motion data and supports single-stage end-to-end training without distillation. The proposed method is evaluated under diverse reference inputs and challenging motion regimes, demonstrating zero-shot transfer to unseen motions as well as robust sim-to-real transfer on a physical humanoid robot.
Paper Structure (26 sections, 15 equations, 5 figures, 2 tables)

This paper contains 26 sections, 15 equations, 5 figures, 2 tables.

Table of Contents

  1. Introduction
  2. METHOD
  3. Humanoid Motion Dataset
  4. Motion Tracking Formulation
  5. Observation Space
  6. Action Space and Low-level Control
  7. Reward Function
  8. Policy Learning Framework
  9. History Encoder
  10. Command Encoder
  11. Fall Recovery Integration
  12. Randomized Recovery Initialization
  13. Termination Conditions
  14. Training Setup
  15. Experiments
  16. ...and 11 more sections

Figures (5)

  • Figure 1: Qualitative results illustrating the generalization of our method across different motion data sources, including public MoCap datasets, video-derived motions obtained from human pose estimation on public videos, and real-time teleoperation demonstrations via VR and a motion-capture suit.
  • Figure 2: Overview of the proposed whole-body control pipeline. A history encoder extracts a dynamics embedding from recent proprioception, which conditions a command encoder to aggregate the contextual command window. The resulting representation is fused with the current observation and fed to an actor-critic policy trained with PPO, and the learned actor is deployed for real-world whole-body motion tracking and teleoperation.
  • Figure 3: Robustness under reference command noise.
  • Figure 4: Real-world dance tracking with fall recovery.
  • Figure 5: Joystick-driven stylized locomotion.