OmniTrack: General Motion Tracking via Physics-Consistent Reference

TL;DR

OmniTrack is introduced, a general tracking framework that explicitly decouples physical feasibility from general motion tracking and supports human-style stable and dynamic online teleoperation, highlighting its robustness and adaptability to varying user inputs.

Abstract

Learning motion tracking from rich human motion data is a foundational task for achieving general control in humanoid robots, enabling them to perform diverse behaviors. However, discrepancies in morphology and dynamics between humans and robots, combined with data noise, introduce physically infeasible artifacts in reference motions, such as floating and penetration. During both training and execution, these artifacts create a conflict between following inaccurate reference motions and maintaining the robot's stability, hindering the development of a generalizable motion tracking policy. To address these challenges, we introduce OmniTrack, a general tracking framework that explicitly decouples physical feasibility from general motion tracking. In the first stage, a privileged generalist policy generates physically plausible motions that strictly adhere to the robot's dynamics via trajectory rollout in simulation. In the second stage, the general control policy is trained to track these physically feasible motions, ensuring stable and coherent control transfer to the real robot. Experiments show that OmniTrack improves tracking accuracy and demonstrates strong generalization to unseen motions. In real-world tests, OmniTrack achieves hour-long, consistent, and stable tracking, including complex acrobatic motions such as flips and cartwheels. Additionally, we show that OmniTrack supports human-style stable and dynamic online teleoperation, highlighting its robustness and adaptability to varying user inputs.

Figures (8)

• Figure 1: Capabilities of OmniTrack in general motion tracking and real-time teleoperation. Leveraging physics-consistent reference motions, OmniTrack achieves general tracking across diverse motion categories, including balance control, high-dynamic maneuvers, and contact-rich interactions. OmniTrack supports real-time teleoperation, enabling the execution of diverse human-style dynamic movements as well as interactive behaviors. These results demonstrate the generality of the framework in handling both physically demanding motions and unstructured motion commands.
• Figure 2: Examples of physically infeasible artifacts in retargeted human motions (blue) compared with physically feasible robot motions (black/gray), including inconsistent center-of-mass motion, foot skating, floating, and ground penetration, which hinder stable humanoid control.
• Figure 3: Overview of the OmniTrack framework. Our method adopts a two-stage pipeline that first converts raw, physically inconsistent reference motions into physics-consistent motions in simulation, and then trains a general policy to robustly track these motions under realistic conditions. The resulting system supports both offline motion tracking and online teleoperation.
• Figure 4: Impact of physically plausible motions under varying dataset sizes. From left to right: mean reward and mean episode length during training, followed by success rate and tracking error (MPJPE) under different dataset sizes (1/8, 1/2, and full LAFAN1). Dark colors denote training with physically plausible motions, while light colors denote training with raw reference motions.
• Figure 5: Diverse motion skills executed on the real humanoid robot. Our policy enables hour-long continuous and stable tracking of a wide range of human-like behaviors, demonstrating broad motion coverage, strong real-world versatility, and long-term control stability.