Proactive Scene Decomposition and Reconstruction

TL;DR

This work tackles the challenge of dynamic scene understanding by introducing proactive scene decomposition and reconstruction as an online task that leverages hand-object interactions observed in egocentric RGB-D streams. The authors propose an online dynamic SLAM system that progressively decomposes the scene into a background and independently moving object instances, guided by interaction priors and rendered with Gaussian splatting for photorealistic results. Key contributions include a: (i) prompted segmentation pipeline that integrates depth-consistency cues with interactive priors, (ii) online camera and object pose estimation coupled with dense map densification, (iii) segmentation refinement and memory-driven strategies to maintain coherent object masks, and (iv) progressive decoupling and joint optimization that update the decomposed map as new interactions occur. Experiments on HOI4D and a new MHOI dataset demonstrate accurate object-level decomposition, robust tracking, and high-quality novel-view rendering, highlighting the method’s potential for robotics, photorealistic simulation, and AR/VR applications. Overall, the approach provides a flexible, interactive, and online alternative to static 4D reconstruction and static-SLAM methods, enabling reliable dynamic scene modeling in real time.

Abstract

Human behaviors are the major causes of scene dynamics and inherently contain rich cues regarding the dynamics. This paper formalizes a new task of proactive scene decomposition and reconstruction, an online approach that leverages human-object interactions to iteratively disassemble and reconstruct the environment. By observing these intentional interactions, we can dynamically refine the decomposition and reconstruction process, addressing inherent ambiguities in static object-level reconstruction. The proposed system effectively integrates multiple tasks in dynamic environments such as accurate camera and object pose estimation, instance decomposition, and online map updating, capitalizing on cues from human-object interactions in egocentric live streams for a flexible, progressive alternative to conventional object-level reconstruction methods. Aided by the Gaussian splatting technique, accurate and consistent dynamic scene modeling is achieved with photorealistic and efficient rendering. The efficacy is validated in multiple real-world scenarios with promising advantages.

Figures (8)

• Figure 1: We introduce a dynamic SLAM system to tackle the proactive scene decomposition and reconstruction from egocentric live streams (first row). Only objects that are under proactive interactions will be decomposed (second) to maintain a consistent granularity. This fashion leads to photorealistic modeling of the environment (third to fifth rows), enabling progressive scene decomposition (fifth row) and robust object tracking (third and fourth rows).
• Figure 2: Overview of our method. With well-defined decomposition granularity induced by motion, the online system achieves reliable camera tracking and scene reconstruction, allowing progressive decomposition and robust instance tracking.
• Figure 3: Comparison of scene decomposition with Gaussian Grouping Ye2024eccv. Gaussian Grouping relies on the pre-defined segmentation granularity of the foundation models, whereas our method achieves adaptive decomposition given live interaction.
• Figure 4: Comparison of segmentation results between SAM2 sam2 and our method given instances under proactive interactions.
• Figure 5: Qualitative comparison of rendering results on HOI4D dataset.