MOGRAS: Human Motion with Grasping in 3D Scenes

TL;DR

The paper tackles the challenge of generating realistic full-body motion with object grasping in cluttered 3D scenes by introducing the MOGRAS dataset, an automated pipeline that provides pre-grasp walking and final grasp poses within richly annotated indoor scenes. It benchmarks existing grasping methods on scene-aware data and proposes GNet++, a scene-conditioned extension of GNet that uses a ViT-based BEV scene embedding and a penetration loss to minimize collisions. The authors demonstrate significant reductions in scene and object penetration and improvements in contact accuracy, validated by quantitative metrics and a human study. Overall, MOGRAS and GNet++ enable more realistic, context-aware human-scene interactions, with broad implications for robotics, animation, and immersive virtual environments.

Abstract

Generating realistic full-body motion interacting with objects is critical for applications in robotics, virtual reality, and human-computer interaction. While existing methods can generate full-body motion within 3D scenes, they often lack the fidelity for fine-grained tasks like object grasping. Conversely, methods that generate precise grasping motions typically ignore the surrounding 3D scene. This gap, generating full-body grasping motions that are physically plausible within a 3D scene, remains a significant challenge. To address this, we introduce MOGRAS (Human MOtion with GRAsping in 3D Scenes), a large-scale dataset that bridges this gap. MOGRAS provides pre-grasping full-body walking motions and final grasping poses within richly annotated 3D indoor scenes. We leverage MOGRAS to benchmark existing full-body grasping methods and demonstrate their limitations in scene-aware generation. Furthermore, we propose a simple yet effective method to adapt existing approaches to work seamlessly within 3D scenes. Through extensive quantitative and qualitative experiments, we validate the effectiveness of our dataset and highlight the significant improvements our proposed method achieves, paving the way for more realistic human-scene interactions.

Figures (6)

• Figure 1: Overview of the MOGRAS Dataset. Our MOGRAS dataset provides full-body motion sequences, including both pre-grasp walking and detailed grasping poses, within richly annotated 3D scenes. This unique combination addresses the limitations of existing datasets, bridging the gap between synthetic data and real-world scenarios to enable realistic and complex human-object interactions.
• Figure 2: Examples from the MOGRAS dataset. MOGRAS provides full-body motion sequences, including a pre-grasp walking phase and a final grasping pose, all within richly annotated 3D scenes. The dataset captures subtle, physically plausible interactions with small objects and their environment, addressing a key gap in existing research.
• Figure 3: Comparison of Original and Refined ScanNet Scenes. Original scenes (pink) often have floor misalignment, causing artifacts like floating feet. Our refined scenes (green) correct this, improving physical plausibility and interaction quality.
• Figure 4: GNet++ Architecture Overview. We extend the GNet architecture to create GNet++ by injecting a scene embedding into both the encoder and decoder, enabling it to generate context-aware grasps that avoid collisions.
• Figure 5: Qualitative Comparison of Generated Grasps. We visualize grasp poses from GNet++ (blue), GOAL (green), and SAGA (pink). In these cluttered scenes, both GOAL and SAGA produce significant penetration with the table and surrounding objects. In contrast, GNet++ successfully generates a physically plausible and collision-free pose by adapting the body to the environment.