HumanX: Toward Agile and Generalizable Humanoid Interaction Skills from Human Videos

TL;DR

HumanX tackles the problem of learning agile and generalizable humanoid interaction skills from scarce realistic data by coupling XGen, a physically grounded data synthesis pipeline that turns monocular human videos into diverse, augmentable humanoid interaction trajectories, with XMimic, a two-stage teacher-student imitation framework that distills diverse, video-derived interactions into a deployable policy. The core contributions are a data generation approach that preserves physical plausibility over photorealism, a unified HOI imitation reward, and a training regimen including disturbance, domain randomization, and perception-agnostic deployment. The method demonstrates ten skills across five domains and zero-shot transfer to a real Unitree G1, achieving over eightfold improvement in generalization over prior HOI imitation methods and enabling complex emergent behaviors in real-time human-robot interaction. Overall, HumanX provides a scalable, task-agnostic pathway to acquire versatile, real-world humanoid interaction skills from human video with strong sim-to-real performance.

Abstract

Enabling humanoid robots to perform agile and adaptive interactive tasks has long been a core challenge in robotics. Current approaches are bottlenecked by either the scarcity of realistic interaction data or the need for meticulous, task-specific reward engineering, which limits their scalability. To narrow this gap, we present HumanX, a full-stack framework that compiles human video into generalizable, real-world interaction skills for humanoids, without task-specific rewards. HumanX integrates two co-designed components: XGen, a data generation pipeline that synthesizes diverse and physically plausible robot interaction data from video while supporting scalable data augmentation; and XMimic, a unified imitation learning framework that learns generalizable interaction skills. Evaluated across five distinct domains--basketball, football, badminton, cargo pickup, and reactive fighting--HumanX successfully acquires 10 different skills and transfers them zero-shot to a physical Unitree G1 humanoid. The learned capabilities include complex maneuvers such as pump-fake turnaround fadeaway jumpshots without any external perception, as well as interactive tasks like sustained human-robot passing sequences over 10 consecutive cycles--learned from a single video demonstration. Our experiments show that HumanX achieves over 8 times higher generalization success than prior methods, demonstrating a scalable and task-agnostic pathway for learning versatile, real-world robot interactive skills.

Figures (11)

• Figure 1: Overview of XGen. The pipeline begins by estimating SMPL‑based human motion from video and retargeting it to the humanoid’s morphology. The video is segmented into contact and non‑contact phases. For the contact phase, a predefined anchor (e.g., the midpoint between the two palms) is used. The object mesh and its relative pose to the anchor are estimated from a keyframe (or defined manually). The object trajectory is then generated by transforming the object according to the anchor’s pose over time, followed by force‑closure optimization to refine the robot poses. During the non‑contact phases, diverse and physically plausible object trajectories are generated via simulation. Complete interaction trajectories are obtained by concatenating and smoothly interpolating the phases. Key steps supporting data augmentation—including object shape and trajectory variation—are highlighted in yellow in the figure.
• Figure 2: Data Augmentation for Contact Phase.
• Figure 3: Data Augmentation for Non-Contact Phase.
• Figure 4: XMimic follows a two‑stage training pipeline. In the Stage 1, a teacher policy is learned with privileged state information under a unified interaction‑imitation reward. In Stage 2, the teacher is distilled into a student policy that operates under realistic perceptual constraints, combining interaction imitation with behavior cloning. The resulting student policy can be deployed directly in real‑world settings.
• Figure 5: Simulation Results on Basketball Catch-Shot. XMimic generalizes to novel ball‑passing trajectories and target positions (green sphere) with accurate and natural interactions.