TWIST: Teleoperated Whole-Body Imitation System

TL;DR

TWIST presents a unified, real-time framework for teleoperating humanoid robots via whole-body motion imitation. It combines large-scale MoCap data with a two-stage teacher-student RL+BC training approach to learn a single controller that faithfully tracks retargeted human motions while maintaining balance. The method demonstrates versatile real-world capabilities across manipulation, locomotion, and expressive actions, with robust performance aided by online retargeting refinements and domain randomization. This work advances general-purpose humanoid teleoperation by enabling coordinated, whole-body skills through a single neural network controller and careful sim-to-real transfer practices.

Abstract

Teleoperating humanoid robots in a whole-body manner marks a fundamental step toward developing general-purpose robotic intelligence, with human motion providing an ideal interface for controlling all degrees of freedom. Yet, most current humanoid teleoperation systems fall short of enabling coordinated whole-body behavior, typically limiting themselves to isolated locomotion or manipulation tasks. We present the Teleoperated Whole-Body Imitation System (TWIST), a system for humanoid teleoperation through whole-body motion imitation. We first generate reference motion clips by retargeting human motion capture data to the humanoid robot. We then develop a robust, adaptive, and responsive whole-body controller using a combination of reinforcement learning and behavior cloning (RL+BC). Through systematic analysis, we demonstrate how incorporating privileged future motion frames and real-world motion capture (MoCap) data improves tracking accuracy. TWIST enables real-world humanoid robots to achieve unprecedented, versatile, and coordinated whole-body motor skills--spanning whole-body manipulation, legged manipulation, locomotion, and expressive movement--using a single unified neural network controller. Our project website: https://humanoid-teleop.github.io

Figures (9)

• Figure 1: The Teleoperated Whole-Body Imitation System (TWIST) is a system that teleoperates humanoid robots with real-time whole-body human data and a single neural network controller. TWIST achieves versatile, coordinated, whole-body skills that are not present in previous works.
• Figure 2: The Teleoperated Whole-Body Imitation System (TWIST) presents versatile, coordinated, and human-like whole-body skills on real-world humanoid robots. Our robot can perform whole-body manipulation (e.g., lifting boxes from the ground), legged manipulation (e.g., kicking the football), locomotion (e.g., walking sideways), and expressive motions (e.g., Waltz dance). More videos: https://humanoid-teleop.github.io
• Figure 3: The Teleoperated Whole-Body Imitation System (TWIST) consists of 3 stages: 1) curating a humanoid motion dataset by retargeting Internet human data and our MoCap data, 2) training a single whole-body controller in simulation, 3) teleoperating real-world humanoid robots with MoCap devices.
• Figure 4: Booster T1 sim2sim results. The whole-body controller is trained in IsaacGym makoviychuk2021isaac and evaluated in MuJoCo todorov2012mujoco. The tracking goals are sampled from training data.
• Figure 5: Teleoperation delay is roughly measured by the video, around 0.9 seconds.