Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Zero-shot Reconstruction of In-Scene Object Manipulation from Video

Dixuan Lin, Tianyou Wang, Zhuoyang Pan, Yufu Wang, Lingjie Liu, Kostas Daniilidis

TL;DR

This work tackles the problem of reconstructing scene-aligned hand–object manipulation from monocular RGB video, addressing the ill-posed nature of scene reconstruction and depth ambiguity. It introduces a zero-shot pipeline that initializes hand pose, object mesh/pose, and scene using foundation models, followed by a two-stage optimization to obtain a metrically consistent hand–object trajectory spanning grasping and interaction in the scene frame. Key contributions include scene-aware reconstruction in world coordinates, a contact- and SDF-constrained interaction optimization, and motion completion via a human motion prior (EgoAllo), with strong results on DexYCB, HOI4D, and in-the-wild footage. The approach advances practical robotic manipulation, AR/VR alignment, and policy learning by delivering reliable, scene-consistent, 3D hand–object motion from single-view videos.

Abstract

We build the first system to address the problem of reconstructing in-scene object manipulation from a monocular RGB video. It is challenging due to ill-posed scene reconstruction, ambiguous hand-object depth, and the need for physically plausible interactions. Existing methods operate in hand centric coordinates and ignore the scene, hindering metric accuracy and practical use. In our method, we first use data-driven foundation models to initialize the core components, including the object mesh and poses, the scene point cloud, and the hand poses. We then apply a two-stage optimization that recovers a complete hand-object motion from grasping to interaction, which remains consistent with the scene information observed in the input video.

Zero-shot Reconstruction of In-Scene Object Manipulation from Video

TL;DR

This work tackles the problem of reconstructing scene-aligned hand–object manipulation from monocular RGB video, addressing the ill-posed nature of scene reconstruction and depth ambiguity. It introduces a zero-shot pipeline that initializes hand pose, object mesh/pose, and scene using foundation models, followed by a two-stage optimization to obtain a metrically consistent hand–object trajectory spanning grasping and interaction in the scene frame. Key contributions include scene-aware reconstruction in world coordinates, a contact- and SDF-constrained interaction optimization, and motion completion via a human motion prior (EgoAllo), with strong results on DexYCB, HOI4D, and in-the-wild footage. The approach advances practical robotic manipulation, AR/VR alignment, and policy learning by delivering reliable, scene-consistent, 3D hand–object motion from single-view videos.

Abstract

We build the first system to address the problem of reconstructing in-scene object manipulation from a monocular RGB video. It is challenging due to ill-posed scene reconstruction, ambiguous hand-object depth, and the need for physically plausible interactions. Existing methods operate in hand centric coordinates and ignore the scene, hindering metric accuracy and practical use. In our method, we first use data-driven foundation models to initialize the core components, including the object mesh and poses, the scene point cloud, and the hand poses. We then apply a two-stage optimization that recovers a complete hand-object motion from grasping to interaction, which remains consistent with the scene information observed in the input video.
Paper Structure (20 sections, 21 equations, 7 figures, 3 tables)

This paper contains 20 sections, 21 equations, 7 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. Hand Pose Estimation
  4. Hand-Object Pose Estimation
  5. Method
  6. Problem formulation
  7. Initial Reconstructions
  8. Hand-Object Motion Optimization
  9. Two Stage Splitting
  10. Interaction Stage Optimization
  11. Grasping Stage Optimization
  12. Experiments
  13. Datasets
  14. Evaluation Metrics
  15. Comparison Results
  16. ...and 5 more sections

Figures (7)

  • Figure 1: We present a zero shot system that reconstruct in-scene object manipulation motion from daily videos.
  • Figure 2: Overview of our framework. We first use foundation models to obtain the hand pose, object, and scene(Sec \ref{['sec:method_recon']}). We then optimize hand–object interaction in the interaction stage by computing contact points and enforcing physical collision constraints(Sec \ref{['sec:inter_stage']}). Finally, in the grasping optimization stage, we complete the motion using a human motion prior and further optimize the approaching and grasping phases (Sec \ref{['sec:grasp_stage']}).
  • Figure 3: (a) Contact candidates. (b) Top: input image. Bottom: the 2D projections of the contact candidates and their intersections with the object mask, where sky blue denotes back-surface contacts and red denotes front-surface contacts. (c) The correspondence between hand and object contact points.
  • Figure 4: Comparison of Egoallo generated motion, interpolation and fullbody motion. The initial hand pose is set to a half-raised posture. Darker color indicates progression over time.(a) shows the hand motion trajectory obtained with EgoAllo. (b) shows the trajectory produced by interpolation. (c) shows the full-body motion trajectory recovered by EgoAllo.
  • Figure 5: Qualitative results on DexYCB. We visualizes our reconstruction of the scene and object in 2 views, with the hand mesh shown in blue. In free view, the baseline hand reconstruction is overlaid in pink for comparison.
  • ...and 2 more figures