4D Gaussian Splatting in the Wild with Uncertainty-Aware Regularization

TL;DR

An uncertainty-aware regularization is introduced that identifies uncertain regions with few observations and selectively imposes additional priors based on diffusion models and depth smoothness on such regions to improve both the performance of novel view synthesis and the quality of training image reconstruction.

Abstract

Novel view synthesis of dynamic scenes is becoming important in various applications, including augmented and virtual reality. We propose a novel 4D Gaussian Splatting (4DGS) algorithm for dynamic scenes from casually recorded monocular videos. To overcome the overfitting problem of existing work for these real-world videos, we introduce an uncertainty-aware regularization that identifies uncertain regions with few observations and selectively imposes additional priors based on diffusion models and depth smoothness on such regions. This approach improves both the performance of novel view synthesis and the quality of training image reconstruction. We also identify the initialization problem of 4DGS in fast-moving dynamic regions, where the Structure from Motion (SfM) algorithm fails to provide reliable 3D landmarks. To initialize Gaussian primitives in such regions, we present a dynamic region densification method using the estimated depth maps and scene flow. Our experiments show that the proposed method improves the performance of 4DGS reconstruction from a video captured by a handheld monocular camera and also exhibits promising results in few-shot static scene reconstruction.

Figures (5)

• Figure 1: Concept of uncertainty-aware regularization. Existing models often use regularization techniques to introduce additional priors for unseen views, aiming to enhance novel view synthesis performance. However, these methods tend to over-regularize accurately reconstructed pixels, which degrades the reconstruction quality of training images. To address this issue, our uncertainty-aware regularization selectively focuses on uncertain regions in unseen views, preserving the quality of well-reconstructed pixels with low uncertainty.
• Figure 2: Visualization of the dynamic region densification on the Backpack scene. Since SfM schonberger2016structure is designed for static scenes, it fails to properly initialize Gaussian primitives in dynamic regions. Our dynamic region densification module initializes additional Gaussian primitives in the identified dynamic regions using scene flow and depth map.
• Figure 3: Qualitative results on the space-out, paper-windmill, teddy, and spin scenes in the DyCheck dataset. UA-4DGS (Ours) shows outstanding quality of rendered images compared to existing methods, including D-3DGS yang2023deformable, Zhan et al.li2023spacetime, and 4DGS wu20234d
• Figure 4: Qualitative comparison between UA-4DGS and other methods tested on the DyCheck dataset. Ours achieves the outstanding quality of rendered images.
• Figure 5: Qualitative comparison between UA-4DGS and other methods tested on the DyCheck dataset. Ours achieves the outstanding quality of rendered images.