Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

World-Coordinate Human Motion Retargeting via SAM 3D Body

Zhangzheng Tu, Kailun Su, Shaolong Zhu, Yukun Zheng

TL;DR

Proposes an engineering-oriented pipeline to recover world-coordinate human motion from monocular video for humanoid-robot retargeting, avoiding SLAM by using SAM 3D Body as a frozen backbone and Momentum Human Rig as a stable intermediate. The method enforces trajectory-level identity and skeleton-scale locking, applies latent-space sliding-window smoothing, and uses a contact-aware global optimization to recover plausible world-root trajectories. It then retargets to Unitree G1 via a two-stage, kinematics-aware IK pipeline, enabling robot-ready motion without heavy temporal models. Experiments on real monocular videos show temporally stable world trajectories and reliable robot retargeting, indicating the practicality of combining structured human representations with lightweight physical constraints.

Abstract

Recovering world-coordinate human motion from monocular videos with humanoid robot retargeting is significant for embodied intelligence and robotics. To avoid complex SLAM pipelines or heavy temporal models, we propose a lightweight, engineering-oriented framework that leverages SAM 3D Body (3DB) as a frozen perception backbone and uses the Momentum HumanRig (MHR) representation as a robot-friendly intermediate. Our method (i) locks the identity and skeleton-scale parameters of per tracked subject to enforce temporally consistent bone lengths, (ii) smooths per-frame predictions via efficient sliding-window optimization in the low-dimensional MHR latent space, and (iii) recovers physically plausible global root trajectories with a differentiable soft foot-ground contact model and contact-aware global optimization. Finally, we retarget the reconstructed motion to the Unitree G1 humanoid using a kinematics-aware two-stage inverse kinematics pipeline. Results on real monocular videos show that our method has stable world trajectories and reliable robot retargeting, indicating that structured human representations with lightweight physical constraints can yield robot-ready motion from monocular input.

World-Coordinate Human Motion Retargeting via SAM 3D Body

TL;DR

Proposes an engineering-oriented pipeline to recover world-coordinate human motion from monocular video for humanoid-robot retargeting, avoiding SLAM by using SAM 3D Body as a frozen backbone and Momentum Human Rig as a stable intermediate. The method enforces trajectory-level identity and skeleton-scale locking, applies latent-space sliding-window smoothing, and uses a contact-aware global optimization to recover plausible world-root trajectories. It then retargets to Unitree G1 via a two-stage, kinematics-aware IK pipeline, enabling robot-ready motion without heavy temporal models. Experiments on real monocular videos show temporally stable world trajectories and reliable robot retargeting, indicating the practicality of combining structured human representations with lightweight physical constraints.

Abstract

Recovering world-coordinate human motion from monocular videos with humanoid robot retargeting is significant for embodied intelligence and robotics. To avoid complex SLAM pipelines or heavy temporal models, we propose a lightweight, engineering-oriented framework that leverages SAM 3D Body (3DB) as a frozen perception backbone and uses the Momentum HumanRig (MHR) representation as a robot-friendly intermediate. Our method (i) locks the identity and skeleton-scale parameters of per tracked subject to enforce temporally consistent bone lengths, (ii) smooths per-frame predictions via efficient sliding-window optimization in the low-dimensional MHR latent space, and (iii) recovers physically plausible global root trajectories with a differentiable soft foot-ground contact model and contact-aware global optimization. Finally, we retarget the reconstructed motion to the Unitree G1 humanoid using a kinematics-aware two-stage inverse kinematics pipeline. Results on real monocular videos show that our method has stable world trajectories and reliable robot retargeting, indicating that structured human representations with lightweight physical constraints can yield robot-ready motion from monocular input.
Paper Structure (21 sections, 8 equations, 1 figure)

This paper contains 21 sections, 8 equations, 1 figure.

Table of Contents

  1. Introduction
  2. Related Work
  3. Evolution of Parametric Human Models: From SMPL to MHR
  4. World-Coordinate Reconstruction from Monocular Human Motion
  5. Method
  6. Preliminaries
  7. Momentum Human Rig (MHR)
  8. SAM 3D Body (3DB)
  9. Video Preprocessing and Tracking
  10. Trajectory-Level Consistency and Latent-Space Smoothing
  11. Trajectory-level shape and scale locking.
  12. Latent-space pose smoothing.
  13. Latent fidelity.
  14. Temporal smoothness.
  15. Window stitching.
  16. ...and 6 more sections

Figures (1)

  • Figure 1: Qualitative results. (a) Temporal consistency of single-person motion estimation. (b) Robust motion estimation under multi-person settings. (c) Ablation on temporal and geometric refinement. Dark blue denotes the current-frame mesh; gray and light blue denote the refined and unrefined predictions two frames ahead, respectively.