Learning Whole-Body Human-Humanoid Interaction from Human-Human Demonstrations

TL;DR

This work tackles the data bottleneck and policy learning challenges in whole-body Human-Humanoid Interaction (HHoI) by introducing PAIR, a Physics-Aware Interaction Retargeting pipeline that preserves contact semantics when retargeting Human-Human Interactions to humanoids, producing physically consistent supervision data. Building on this, the authors propose D-STAR, a Decoupled Spatio-Temporal Action Reasoner that separates when to act (Phase Attention) from where to act (Multi-Scale Spatial), with a diffusion-based planning head that yields high-level targets executed by a robust Whole-Body Controller. The approach yields significant performance gains over baselines in simulation (e.g., average policy success of 75.4% and substantial gains on Handshake) and demonstrates sim-to-real transfer on a Unitree G1 with asynchronous sensing and a standard WBC. Overall, PAIR and D-STAR form a complete, scalable pipeline that moves beyond mimicry toward true interactive intelligence in whole-body HHoI, enabling safe, responsive collaboration with humans in real-world settings.

Abstract

Enabling humanoid robots to physically interact with humans is a critical frontier, but progress is hindered by the scarcity of high-quality Human-Humanoid Interaction (HHoI) data. While leveraging abundant Human-Human Interaction (HHI) data presents a scalable alternative, we first demonstrate that standard retargeting fails by breaking the essential contacts. We address this with PAIR (Physics-Aware Interaction Retargeting), a contact-centric, two-stage pipeline that preserves contact semantics across morphology differences to generate physically consistent HHoI data. This high-quality data, however, exposes a second failure: conventional imitation learning policies merely mimic trajectories and lack interactive understanding. We therefore introduce D-STAR (Decoupled Spatio-Temporal Action Reasoner), a hierarchical policy that disentangles when to act from where to act. In D-STAR, Phase Attention (when) and a Multi-Scale Spatial module (where) are fused by the diffusion head to produce synchronized whole-body behaviors beyond mimicry. By decoupling these reasoning streams, our model learns robust temporal phases without being distracted by spatial noise, leading to responsive, synchronized collaboration. We validate our framework through extensive and rigorous simulations, demonstrating significant performance gains over baseline approaches and a complete, effective pipeline for learning complex whole-body interactions from HHI data.

Figures (6)

• Figure 1: From HHI to HHoI with simulation and real-robot results.Left: PAIR (Physics-Aware Interaction Retargeting) converts human--human interaction sequences into physically consistent human--humanoid (HHoI) clips by aligning morphology and explicitly preserving contact semantics via a two-stage pipeline. Top (Sim): Rollouts of the learned policy (D-STAR) in simulation, showing Bend, Wave, Fly-Kiss, Hug, High-Five, and Handshake, demonstrating synchronized whole-body interactions. Bottom (Real, a--c): Deployment on a Unitree G1 under a standard whole-body controller; the policy executes Hug, Handshake, and High-Five selected via text commands.
• Figure 2: PAIR preserves physical consistency where naive methods fail. Left: A source HHI handshake. Center: Naive retargeting breaks essential contact due to morphological disparities. Right: PAIR first ensures kinematic plausibility, then applies an interaction-aware objective ($\mathcal{L}_{\text{con}}$) to refine and enforce the critical physical contact.
• Figure 3: PAIR preserves contact semantics and physical consistency via a two-stage retargeting pipeline. From HHI to HHoI while retaining contact semantics across morphology differences.
• Figure 4: Overview of D-STAR (Decoupled Spatio-Temporal Action Reasoner): Phase Attention (PA, “when to act”) and Multi-Scale Spatial module (MSS, “where to act”) are fused by a diffusion planning head to yield synchronized whole-body interaction beyond mimicry; a low-level Whole-Body Controller (WBC) executes the final physically plausible action.
• Figure 5: Monocular SMPL perception setup and result. (a) Unitree G1 with a Logitech C1000e RGB camera ($1280\times720@30$ fps). (b) Input RGB frame. (c) SMPL mesh estimated from the single RGB stream (4D-Humans goel2023humans) and mapped to the robot base frame after extrinsic calibration.