Space-time 2D Gaussian Splatting for Accurate Surface Reconstruction under Complex Dynamic Scenes

TL;DR

Problem: reconstruct accurate surfaces in complex dynamic scenes from sparse views with occlusions. Approach: space-time 2D Gaussian Splatting that learns canonical Gaussian splats and deforms them over time with geometry and opacity fields, plus a Hex-Plane deformation and an opacity-modeling strategy, and employs TSDF fusion for mesh extraction. Key contributions: first particle-based surface model for dynamic scenes; joint optimization of canonical and deformed Gaussians; time-varying opacity with foreground-mask regularization; strong empirical gains on CMU Panoptic and D-NeRF with faster training and rendering than prior volumetric or hybrid methods. Impact: enables high-fidelity dynamic surface reconstructions in real-world, multi-person and human-object interaction scenes, with practical efficiency for sparse-view inputs.

Abstract

Previous surface reconstruction methods either suffer from low geometric accuracy or lengthy training times when dealing with real-world complex dynamic scenes involving multi-person activities, and human-object interactions. To tackle the dynamic contents and the occlusions in complex scenes, we present a space-time 2D Gaussian Splatting approach. Specifically, to improve geometric quality in dynamic scenes, we learn canonical 2D Gaussian splats and deform these 2D Gaussian splats while enforcing the disks of the Gaussian located on the surface of the objects by introducing depth and normal regularizers. Further, to tackle the occlusion issues in complex scenes, we introduce a compositional opacity deformation strategy, which further reduces the surface recovery of those occluded areas. Experiments on real-world sparse-view video datasets and monocular dynamic datasets demonstrate that our reconstructions outperform state-of-the-art methods, especially for the surface of the details. The project page and more visualizations can be found at: https://tb2-sy.github.io/st-2dgs/.

Figures (14)

• Figure 1: Given sparse view or monocular video input, our method achieves high-quality surface reconstruction and rendering of dynamic scenes.
• Figure 2: Two different scene representations. SDF-based methods wang2021neusvolsdf capture surfaces at specific locations, while our particle-based representation utilizes discrete 2D Gaussians 2dgs. This particle-based method is more storage-efficient, offers faster rendering, and simplifies motion tracking.
• Figure 3: Different opacity deformation approach for Pizza1 scene where canonical space opacity distribution (smooth results) over training iterations. a) Without opacity deformation and b) Multiplication zhao2024gaussianprediction, the opacity distribution gradually decreases from the larger end, failing to maintain a binary opacity state. c) Additon gs4d results in an unstable distribution. d) Our approach allows canonical 2D Gaussians to maintain a stable and clear binary opacity state.
• Figure 4: In (a), we illustrate the elements of sudden appearance. In (c), we visualize the changes in opacity deformation weight $\gamma$ over time. The points with significant changes in opacity weight are marked in red in (b), which corresponds to the sudden appearance of elements shown in (a). This further demonstrates the effectiveness of our proposed opacity deformation.
• Figure 5: Visual comparisons at two different timestamps between our method, SDFFlow sdfflow, and 4DGS gs4d are conducted using scenes from a real-world dataset cmu. Our method, along with 4DGS, captures more details than the SDFFlow by leveraging the advantages of a point-based approach. However, because 4DGS is an extension of 3DGS, it suffers from insufficient geometric accuracy and produces a noisy surface.