Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Asset-Driven Sematic Reconstruction of Dynamic Scene with Multi-Human-Object Interactions

Sandika Biswas, Qianyi Wu, Biplab Banerjee, Hamid Rezatofighi

TL;DR

This work tackles the problem of reconstructing dynamic, multi-entity scenes from monocular video by introducing Asset-Driven Scene Reconstruction (ADSR), a hybrid pipeline that combines high-fidelity 3D asset generation for scene elements, semantic-aware deformation of humans and rigid objects, and Gaussian Splatting-based refinement. The approach initializes per-object meshes from canonical frames using 3D generative models, deforms non-rigid humans via SMPL-based skinning, tracks rigid objects through depth-informed mappings, and refines the overall scene with GS optimization to achieve temporally coherent, multiview-consistent geometry. Evaluations on the HOI-M3 dataset show improved reconstruction quality over state-of-the-art methods, demonstrating robustness to occlusions and complex interactions. The method advances monocular dynamic scene reconstruction by effectively fusing multi-modal priors with differentiable rendering to produce detailed 4D geometry suitable for AR/VR and embodied AI applications.

Abstract

Real-world human-built environments are highly dynamic, involving multiple humans and their complex interactions with surrounding objects. While 3D geometry modeling of such scenes is crucial for applications like AR/VR, gaming, and embodied AI, it remains underexplored due to challenges like diverse motion patterns and frequent occlusions. Beyond novel view rendering, 3D Gaussian Splatting (GS) has demonstrated remarkable progress in producing detailed, high-quality surface geometry with fast optimization of the underlying structure. However, very few GS-based methods address multihuman, multiobject scenarios, primarily due to the above-mentioned inherent challenges. In a monocular setup, these challenges are further amplified, as maintaining structural consistency under severe occlusion becomes difficult when the scene is optimized solely based on GS-based rendering loss. To tackle the challenges of such a multihuman, multiobject dynamic scene, we propose a hybrid approach that effectively combines the advantages of 1) 3D generative models for generating high-fidelity meshes of the scene elements, 2) Semantic-aware deformation, \ie rigid transformation of the rigid objects and LBS-based deformation of the humans, and mapping of the deformed high-fidelity meshes in the dynamic scene, and 3) GS-based optimization of the individual elements for further refining their alignments in the scene. Such a hybrid approach helps maintain the object structures even under severe occlusion and can produce multiview and temporally consistent geometry. We choose HOI-M3 for evaluation, as, to the best of our knowledge, this is the only dataset featuring multihuman, multiobject interactions in a dynamic scene. Our method outperforms the state-of-the-art method in producing better surface reconstruction of such scenes.

Asset-Driven Sematic Reconstruction of Dynamic Scene with Multi-Human-Object Interactions

TL;DR

This work tackles the problem of reconstructing dynamic, multi-entity scenes from monocular video by introducing Asset-Driven Scene Reconstruction (ADSR), a hybrid pipeline that combines high-fidelity 3D asset generation for scene elements, semantic-aware deformation of humans and rigid objects, and Gaussian Splatting-based refinement. The approach initializes per-object meshes from canonical frames using 3D generative models, deforms non-rigid humans via SMPL-based skinning, tracks rigid objects through depth-informed mappings, and refines the overall scene with GS optimization to achieve temporally coherent, multiview-consistent geometry. Evaluations on the HOI-M3 dataset show improved reconstruction quality over state-of-the-art methods, demonstrating robustness to occlusions and complex interactions. The method advances monocular dynamic scene reconstruction by effectively fusing multi-modal priors with differentiable rendering to produce detailed 4D geometry suitable for AR/VR and embodied AI applications.

Abstract

Real-world human-built environments are highly dynamic, involving multiple humans and their complex interactions with surrounding objects. While 3D geometry modeling of such scenes is crucial for applications like AR/VR, gaming, and embodied AI, it remains underexplored due to challenges like diverse motion patterns and frequent occlusions. Beyond novel view rendering, 3D Gaussian Splatting (GS) has demonstrated remarkable progress in producing detailed, high-quality surface geometry with fast optimization of the underlying structure. However, very few GS-based methods address multihuman, multiobject scenarios, primarily due to the above-mentioned inherent challenges. In a monocular setup, these challenges are further amplified, as maintaining structural consistency under severe occlusion becomes difficult when the scene is optimized solely based on GS-based rendering loss. To tackle the challenges of such a multihuman, multiobject dynamic scene, we propose a hybrid approach that effectively combines the advantages of 1) 3D generative models for generating high-fidelity meshes of the scene elements, 2) Semantic-aware deformation, \ie rigid transformation of the rigid objects and LBS-based deformation of the humans, and mapping of the deformed high-fidelity meshes in the dynamic scene, and 3) GS-based optimization of the individual elements for further refining their alignments in the scene. Such a hybrid approach helps maintain the object structures even under severe occlusion and can produce multiview and temporally consistent geometry. We choose HOI-M3 for evaluation, as, to the best of our knowledge, this is the only dataset featuring multihuman, multiobject interactions in a dynamic scene. Our method outperforms the state-of-the-art method in producing better surface reconstruction of such scenes.

Paper Structure

This paper contains 9 sections, 10 equations, 5 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. Methodology
  4. High-fidelity 3D generation for scene objects:
  5. Temporally-coherent 3D Deformation of humans:
  6. Tracking rigid objects through video frames:
  7. Scene Refinement using Gaussian optimizer:
  8. Experiments
  9. Conclusions

Figures (5)

  • Figure 1: Semantic 3D geometry reconstruction of a dynamic scene with multiple human-object interactions (HOI-M3 zhang2024hoi dataset). Our method can produce detailed, multiview-consistent, better-quality geometry with plausible overall scene reconstruction compared to the state-of-the-art method.
  • Figure 2: Overall pipeline of our method. Given a monocular video, we first generate the 3D mesh representation for every object (Sec 3.1), deform the human mesh (Sec 3.2), and transform the rigid objects (Sec 3.3) to align with every frame, and finally, fine-tune their alignment with GS-based optimization (Sec 3.4) to get a scene-level reconstruction.
  • Figure 3: Deformation of the human mesh, generated by a 3D generative model, from the canonical frame to every other frame in the sequence.
  • Figure 4: Reconstruction results for two scenes, livingroom$\_$data12 and livingroom$\_$data36 of the HOI-M3 dataset zhang2024hoi with multi-human multi-object interactions. For every example, we show the reconstruction results from two different views. Our method can produce detailed, multiview-consistent, better-quality geometry with plausible overall scene reconstruction.
  • Figure 5: (a) Input image, (b), (c) An instance of object alignments before and after Gaussian Splatting-based optimization. Areas under the marked rectangles show misalignment between the scene elements before GS-based optimization, which improves with GS-based optimization.