Relaxing Accurate Initialization Constraint for 3D Gaussian Splatting

TL;DR

This work addresses the dependence of 3D Gaussian Splatting (3DGS) on accurate initialization by introducing RAIN-GS, a strategy that relaxes initialization requirements through sparse-large-variance initialization, progressive Gaussian low-pass filtering, and adaptive bound-expanding splits. Frequency-domain analysis of SfM initialization reveals that low-frequency components guide a coarse-to-fine learning trajectory, which RAIN-GS emulates even with sub-optimal starts. The approach enables robust training from random or noisy initial point clouds, delivering results on par with or better than SfM-based 3DGS across multiple datasets, with demonstrated improvements via ablations. Overall, RAIN-GS broadens the applicability of 3DGS to scenarios where obtaining an accurate initial point cloud is challenging, while maintaining real-time rendering capabilities and offering practical benefits for real-world 3D reconstruction and view synthesis.

Abstract

3D Gaussian splatting (3DGS) has recently demonstrated impressive capabilities in real-time novel view synthesis and 3D reconstruction. However, 3DGS heavily depends on the accurate initialization derived from Structure-from-Motion (SfM) methods. When the quality of the initial point cloud deteriorates, such as in the presence of noise or when using randomly initialized point cloud, 3DGS often undergoes large performance drops. To address this limitation, we propose a novel optimization strategy dubbed RAIN-GS (Relaing Accurate Initialization Constraint for 3D Gaussian Splatting). Our approach is based on an in-depth analysis of the original 3DGS optimization scheme and the analysis of the SfM initialization in the frequency domain. Leveraging simple modifications based on our analyses, RAIN-GS successfully trains 3D Gaussians from sub-optimal point cloud (e.g., randomly initialized point cloud), effectively relaxing the need for accurate initialization. We demonstrate the efficacy of our strategy through quantitative and qualitative comparisons on multiple datasets, where RAIN-GS trained with random point cloud achieves performance on-par with or even better than 3DGS trained with accurate SfM point cloud. Our project page and code can be found at https://ku-cvlab.github.io/RAIN-GS.

Figures (16)

• Figure 1: Effectiveness of our simple strategy.Left and right show the results from 3DGS kerbl20233d and ours trained with randomly initialized point cloud respectively. Transition from 3DGS to ours simply requires our strategy consisted of sparse-large-variance (SLV) random initialization, progressive Gaussian low-pass filtering, and adaptive bound-expanding split (ABE-Split) algorithm.
• Figure 2: Analysis of SfM initialization in 3DGS. (a) The top shows the GT image, and the bottom is the rendered image by 3DGS after only 10 steps with SfM initialization. We can observe that the rendered image is already coarsely-close to GT image. We randomly sample a horizontal line from the image marked in red. (b) The pixel intensity along this line are shown, with the GT indicated in blue and the rendered image in orange. (c) This graph visualizes the magnitude of the frequency components of (b). Since frequencies further from the middle of the x-axis represent high-frequency components, we observe that SfM provides coarse approximation of the true distribution.
• Figure 3: Toy experiment to analyze different initialization methods. This figure visualizes the result of our toy experiment predicting the target distribution using a collection of 1D Gaussians, starting from different initialization methods.
• Figure 4: Qualitative results on Mip-NeRF360, Tanks&Temples, and Deep Blending dataset.
• Figure 5: Visualization of different initialization methods. This figure illustrates the effect of different initialization methods. (a) Ground truth image. (b) Initialized point cloud from Structure-from-Motion (SfM). (c) Dense-small-variance (DSV) random initialization with small initial covariances due to the short distance between Gaussians. (d) Sparse-large-variance (SLV) random initialization with large initial covariances due to wider distance between Gaussians.