CLOT: Closed-Loop Global Motion Tracking for Whole-Body Humanoid Teleoperation

TL;DR

CLOT tackles the challenge of drift in long-horizon humanoid teleoperation by introducing closed-loop global pose tracking with high-frequency localization feedback. The method combines a data-driven Observation Pre-shift strategy, a Transformer-based PPO policy, and an adversarial motion prior to achieve smooth, global corrections while avoiding aggressive, unsafe maneuvers. A 20-hour human motion dataset and real-time IK retargeting enable robust sim-to-real transfer, demonstrated on a full-size 31-DoF humanoid (Adam Pro) with strong disturbance rejection and high-precision loco-manipulation. The work advances drift-free, high-dynamic teleoperation and provides a practical framework for robust, long-duration humanoid control in real-world settings.

Abstract

Long-horizon whole-body humanoid teleoperation remains challenging due to accumulated global pose drift, particularly on full-sized humanoids. Although recent learning-based tracking methods enable agile and coordinated motions, they typically operate in the robot's local frame and neglect global pose feedback, leading to drift and instability during extended execution. In this work, we present CLOT, a real-time whole-body humanoid teleoperation system that achieves closed-loop global motion tracking via high-frequency localization feedback. CLOT synchronizes operator and robot poses in a closed loop, enabling drift-free human-to-humanoid mimicry over long timehorizons. However, directly imposing global tracking rewards in reinforcement learning, often results in aggressive and brittle corrections. To address this, we propose a data-driven randomization strategy that decouples observation trajectories from reward evaluation, enabling smooth and stable global corrections. We further regularize the policy with an adversarial motion prior to suppress unnatural behaviors. To support CLOT, we collect 20 hours of carefully curated human motion data for training the humanoid teleoperation policy. We design a transformer-based policy and train it for over 1300 GPU hours. The policy is deployed on a full-sized humanoid with 31 DoF (excluding hands). Both simulation and real-world experiments verify high-dynamic motion, high-precision tracking, and strong robustness in sim-to-real humanoid teleoperation. Motion data, demos and code can be found in our website.

Figures (10)

• Figure 1: Key parameters of PNDbotics Adam Pro.
• Figure 2: Overview of the CLOT pipeline.Phase 1: Data Pipeline. Human motion is captured using a hybrid optical--inertial motion capture system and retargeted to the humanoid kinematic motion via IK-based retargeting. Phase 2: RL Training Framework. We propose a random observation pre-shift strategy that decouples policy observations from reward-aligned references, enabling implicit learning of smooth motion interpolation and compliant global corrections. A Transformer policy models long-horizon dependencies, while an adversarial motion prior (AMP) enforces motion realism. Simulation training was performed in mjlab mjlab. Phase 3: Sim-to-Real Development. Real-time retargeted operator motions provide reference commands, while high-frequency global pose estimates of the robot close the feedback loop, enabling stable and drift-free long-horizon teleoperation.
• Figure 3: Samples from our captured motion dataset. The dataset includes diverse motions, from locomotion to high-dynamic whole-body behaviors.
• Figure 4: Motion capture cameras and marker points. Our motion capture system obtains the robot’s global pose by tracking the positions of the marker points.
• Figure 5: Real-world teleoperated loco-manipulation experiments on PNDbotics Adam Pro. The sequential progression of each task is illustrated from (a) to (e). Throughout the execution, the humanoid robot maintains a constant spatial offset relative to the operator's movements in the world frame. All robot motions across all tasks was governed by a single operator, demonstrating the system's efficiency in whole-body teleoperation.