Evaluating Alternatives to SFM Point Cloud Initialization for Gaussian Splatting

TL;DR

This work addresses the bottleneck of SFM-based initialization for Gaussian Splatting by exploring lower-cost alternatives. It demonstrates that carefully designed random initialization, when combined with structure guidance from short-duration NeRF models and depth distillation, can match or surpass COLMAP initialization, enabling COLMAP-free training pipelines. The study shows substantial runtime savings in SLAM-based camera-estimation setups and establishes a practical pathway for scalable, real-time 3D scene reconstruction and novel-view synthesis. The findings highlight the utility of NeRF-derived priors for bootstrapping geometric priors in rasterized Gaussian representations, with broad implications for robotics and vision applications where full SFM is prohibitive.

Abstract

3D Gaussian Splatting has recently been embraced as a versatile and effective method for scene reconstruction and novel view synthesis, owing to its high-quality results and compatibility with hardware rasterization. Despite its advantages, Gaussian Splatting's reliance on high-quality point cloud initialization by Structure-from-Motion (SFM) algorithms is a significant limitation to be overcome. To this end, we investigate various initialization strategies for Gaussian Splatting and delve into how volumetric reconstructions from Neural Radiance Fields (NeRF) can be utilized to bypass the dependency on SFM data. Our findings demonstrate that random initialization can perform much better if carefully designed and that by employing a combination of improved initialization strategies and structure distillation from low-cost NeRF models, it is possible to achieve equivalent results, or at times even superior, to those obtained from SFM initialization. Source code is available at https://theialab.github.io/nerf-3dgs .

Figures (6)

• Figure 1: COLMAP schoenberger2016sfmschoenberger2016mvs (Left) vs. NeRF-based (Right) initialization point clouds for Gaussian Splatting. The NeRF-based initialization provides a much more complete model of the scene structure, while also being faster to construct with posed images.
• Figure 2: COLMAP (Left) vs. NeRF-based (Right) initialization point clouds for Gaussian Splatting. In this scene from OMMO lu2023large (Top), the surface of the water is dominated by dynamic content (waves), and view-dependent effects (reflections), which causes the SFM point cloud to be nearly empty in these areas. In contrast, the NeRF initialization is able to place points near this surface despite the lack of view-consistent detail. Even for more static and Lambertian scenes, such as the garden from Mip-NeRF 360 barron2022mip (Bottom), the NeRF point cloud is still significantly more complete.
• Figure 3: Qualitative results on a scene from the OMMO dataset lu2023large. We find that the COLMAP and random initializations allocate fewer points to the surface of the water, which leads to over-smoothed reconstruction compared to the NeRF-based initialization and models with depth supervision. Please zoom in to see details. Please also refer to the interactive comparison in the supplementary material.
• Figure 4: Additional qualitative results from the OMMO dataset lu2023large and Mip-NeRF 360 barron2022mip. Here we visualize only the result of training with COLMAP and our best performing model with NeRF initialization + depth supervision, to highlight the total contribution of the strategies we evaluated. Please zoom in to see details.
• Figure 5: Qualitative results on a scene from the Mip-NeRF 360 dataset lu2023large. We observe that the denser initialization from NeRF, and the structure guidance both lead to decreased loss of detail and thin geometry. Please zoom in to see details. Please also refer to the interactive comparison in the supplementary material.