Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Learning Athletic Humanoid Tennis Skills from Imperfect Human Motion Data

Zhikai Zhang, Haofei Lu, Yunrui Lian, Ziqing Chen, Yun Liu, Chenghuai Lin, Han Xue, Zicheng Zeng, Zekun Qi, Shaolin Zheng, Qing Luan, Jingbo Wang, Junliang Xing, He Wang, Li Yi

Abstract

Human athletes demonstrate versatile and highly-dynamic tennis skills to successfully conduct competitive rallies with a high-speed tennis ball. However, reproducing such behaviors on humanoid robots is difficult, partially due to the lack of perfect humanoid action data or human kinematic motion data in tennis scenarios as reference. In this work, we propose LATENT, a system that Learns Athletic humanoid TEnnis skills from imperfect human motioN daTa. The imperfect human motion data consist only of motion fragments that capture the primitive skills used when playing tennis rather than precise and complete human-tennis motion sequences from real-world tennis matches, thereby significantly reducing the difficulty of data collection. Our key insight is that, despite being imperfect, such quasi-realistic data still provide priors about human primitive skills in tennis scenarios. With further correction and composition, we learn a humanoid policy that can consistently strike incoming balls under a wide range of conditions and return them to target locations, while preserving natural motion styles. We also propose a series of designs for robust sim-to-real transfer and deploy our policy on the Unitree G1 humanoid robot. Our method achieves surprising results in the real world and can stably sustain multi-shot rallies with human players. Project page: https://zzk273.github.io/LATENT/

Learning Athletic Humanoid Tennis Skills from Imperfect Human Motion Data

Abstract

Human athletes demonstrate versatile and highly-dynamic tennis skills to successfully conduct competitive rallies with a high-speed tennis ball. However, reproducing such behaviors on humanoid robots is difficult, partially due to the lack of perfect humanoid action data or human kinematic motion data in tennis scenarios as reference. In this work, we propose LATENT, a system that Learns Athletic humanoid TEnnis skills from imperfect human motioN daTa. The imperfect human motion data consist only of motion fragments that capture the primitive skills used when playing tennis rather than precise and complete human-tennis motion sequences from real-world tennis matches, thereby significantly reducing the difficulty of data collection. Our key insight is that, despite being imperfect, such quasi-realistic data still provide priors about human primitive skills in tennis scenarios. With further correction and composition, we learn a humanoid policy that can consistently strike incoming balls under a wide range of conditions and return them to target locations, while preserving natural motion styles. We also propose a series of designs for robust sim-to-real transfer and deploy our policy on the Unitree G1 humanoid robot. Our method achieves surprising results in the real world and can stably sustain multi-shot rallies with human players. Project page: https://zzk273.github.io/LATENT/
Paper Structure (39 sections, 4 equations, 8 figures, 5 tables)

This paper contains 39 sections, 4 equations, 8 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Latent Action Space for Humanoid Control
  4. Humanoid for Sports
  5. Learning Tennis Skills with LATENT
  6. Collection of Imperfect Human Motion Data
  7. Correctable Latent Action Space Construction
  8. Motion tracker Pre-training
  9. Online Distillation with Variational Bottleneck
  10. High-Level Policy Learning
  11. Task Setting
  12. Action Correction
  13. Latent Action Barrier
  14. Sim-to-Real Transfer
  15. Dynamics Randomization
  16. ...and 24 more sections

Figures (8)

  • Figure 1: (a) The humanoid performs multi-shot rallies with a human player using different stroke types across various court regions. (b) The humanoid performs athletic tennis skills to strike an incoming ball traveling at high speed (peak velocities > 15 m/s).
  • Figure 2: Overview of LATENT. (a) We pre-train a motion tracker on collected imperfect human motion data. (b) We construct a correctable latent action space via online distillation. (c) We train a high-level policy to correct and compose latent actions for tennis task. (d) We transfer the policy to the real world via dynamics randomization and observation noise.
  • Figure 3: Visualization of generated ball trajectories.
  • Figure 4: The motivation of Latent Action Barrier (LAB).
  • Figure 5: Robot movement coverage during consecutive ball returns. Heatmaps of the robot's global positions accumulated over different numbers of consecutive ball returns (8, 16, 80, and 400). The learned policy enables effective court coverage and adaptive repositioning during consecutive rallies.
  • ...and 3 more figures