MOSAIC-GS: Monocular Scene Reconstruction via Advanced Initialization for Complex Dynamic Environments

TL;DR

MOSAIC-GS tackles the ill-posed problem of monocular dynamic scene reconstruction by introducing an explicit, Gaussian Splatting-based framework that disentangles static and dynamic components and relies on a robust initialization stage. It leverages multi-cue pre-processing, including dynamic region detection, segmentation/tracking, and scene flow refinement to initialize time-parameterized Poly-Fourier trajectories for dynamic Gaussians, significantly speeding up training and rendering. The method achieves competitive PSNR and state-of-the-art LPIPS on standard benchmarks, while offering dynamic segmentation and editing capabilities with no extra computation. This combination of accurate non-rigid motion handling, compact representation, and fast inference holds meaningful practical impact for AR/VR, robotics, and 3D content creation using monocular video data.

Abstract

We present MOSAIC-GS, a novel, fully explicit, and computationally efficient approach for high-fidelity dynamic scene reconstruction from monocular videos using Gaussian Splatting. Monocular reconstruction is inherently ill-posed due to the lack of sufficient multiview constraints, making accurate recovery of object geometry and temporal coherence particularly challenging. To address this, we leverage multiple geometric cues, such as depth, optical flow, dynamic object segmentation, and point tracking. Combined with rigidity-based motion constraints, these cues allow us to estimate preliminary 3D scene dynamics during an initialization stage. Recovering scene dynamics prior to the photometric optimization reduces reliance on motion inference from visual appearance alone, which is often ambiguous in monocular settings. To enable compact representations, fast training, and real-time rendering while supporting non-rigid deformations, the scene is decomposed into static and dynamic components. Each Gaussian in the dynamic part of the scene is assigned a trajectory represented as time-dependent Poly-Fourier curve for parameter-efficient motion encoding. We demonstrate that MOSAIC-GS achieves substantially faster optimization and rendering compared to existing methods, while maintaining reconstruction quality on par with state-of-the-art approaches across standard monocular dynamic scene benchmarks.

Figures (7)

• Figure 1: MOSAIC-GS produces high-quality reconstructions of dynamic scenes, capturing fine details in dynamic regions more accurately while significantly reducing training and rendering time compared to prior methods. Beyond reconstruction, our approach enables dynamic scene segmentation and editing, allowing objects to be removed, isolated, or manipulated without any additional computational cost.
• Figure 2: Overview of the MOSAIC-GS framework. The pipeline begins with four pre-processing steps: (1) dynamic region detection using optical flow and epipolar error, (2) segmentation and tracking of dynamic instances, (3) scene flow estimation with per-instance rigidity refinement, and (4) initialization of static and dynamic Gaussians with Poly-Fourier encoded trajectories. The extracted parameters serve as initialization for the main photometric optimization phase, where separate Gaussian models are created for static and dynamic regions to improve parameter efficiency, while being jointly rasterized for rendering. A depth-based Pearson correlation loss further enhances geometric consistency and reconstruction fidelity.
• Figure 3: Dynamic object detection and tracking. Masks of previously detected objects are subtracted from dynamic regions to identify new instances. The remaining regions are used as prompts for SAM2 ravi2024sam2 to generate segmentation masks, which are tracked forward through all frames. A reverse propagation pass extends object masks to earlier frames where the object was initially static.
• Figure 4: Qualitative comparison of novel view synthesis on the DyCheck gao2022dynamic dataset. Our method produces sharper details in dynamic regions compared to prior approaches resulting in higher perceptual similarity with the ground truth evaluation view.
• Figure 5: Qualitative comparison of novel view synthesis on the NVIDIA Dynamic Scene (original) dataset. The proposed method better captures motion and preserves objects' geometry when observed from different viewpoints.