EgoGaussian: Dynamic Scene Understanding from Egocentric Video with 3D Gaussian Splatting

TL;DR

EgoGaussian tackles dynamic scene understanding from RGB egocentric video by separating static background from interacting objects and modeling them with 3D Gaussian Splatting. Through a clip-level online learning pipeline, it first reconstructs the static background and initializes object Gaussians, then progressively refines dynamic object shapes and estimates per-frame rigid poses to capture 4D scene evolution. The method outperforms state-of-the-art dynamic 3D Gaussian approaches on HOI4D and EPIC-KITCHENS for novel-view synthesis and dynamic object reconstruction, while offering interpretable object-aware representations and motion trajectories. This work advances practical 4D reconstruction from monocular RGB egocentric data, with potential applications in AR, robotics, and immersive scene understanding.

Abstract

Human activities are inherently complex, often involving numerous object interactions. To better understand these activities, it is crucial to model their interactions with the environment captured through dynamic changes. The recent availability of affordable head-mounted cameras and egocentric data offers a more accessible and efficient means to understand human-object interactions in 3D environments. However, most existing methods for human activity modeling neglect the dynamic interactions with objects, resulting in only static representations. The few existing solutions often require inputs from multiple sources, including multi-camera setups, depth-sensing cameras, or kinesthetic sensors. To this end, we introduce EgoGaussian, the first method capable of simultaneously reconstructing 3D scenes and dynamically tracking 3D object motion from RGB egocentric input alone. We leverage the uniquely discrete nature of Gaussian Splatting and segment dynamic interactions from the background, with both having explicit representations. Our approach employs a clip-level online learning pipeline that leverages the dynamic nature of human activities, allowing us to reconstruct the temporal evolution of the scene in chronological order and track rigid object motion. EgoGaussian shows significant improvements in terms of both dynamic object and background reconstruction quality compared to the state-of-the-art. We also qualitatively demonstrate the high quality of the reconstructed models.

Figures (7)

• Figure 1: EgoGaussian Pipeline. Given an egocentric video input, our framework first partitions the video input into static and dynamic clips and obtains hand-object segmentation masks using off-the-shelf approaches. The static clips are then used to reconstruct the background scenes and initialize the shapes of the object that will be interacted with using 3D Gaussians. Subsequently, we refine the object’s shapes and track their motion through the dynamic clips. EgoGaussian enables a high-quality 4D reconstruction of dynamic scenes with explicit representations that accurately capture the object interactions.
• Figure 2: Dynamic Object Modeling Pipeline. We use a sequential pipeline with regularization from previous frames, allowing us to estimate object poses and iteratively refine their shapes simultaneously.
• Figure 3: Qualitative comparison with SOTA. We show reconstructions produced by our method and SOTA baselines (4DGS 4DGaussianSplatting and Deformable 3DGS Deformable3DGaussians) along with their modified versions from both HOI4D and EPIC-KITCHENS. Our reconstruction demonstrates more accurate reconstructions while baseline approaches fail to handle dynamic interactions.
• Figure 4: More qualitative results of the reconstructed dynamic scenes. The left figure shows the object trajectory inferred from interpolated object poses. The right figure shows the rendering from arbitrary viewpoints.
• Figure 5: Example of black artifacts