Initialize to Generalize: A Stronger Initialization Pipeline for Sparse-View 3DGS

TL;DR

Sparse-view 3D Gaussian Splatting (3DGS) overfits to training views, causing artifacts in novel views. The authors conduct controlled ablations to show initialization quality sets the achievable performance band, and training-time constraints offer only modest gains. They propose a three-stage initialization pipeline—low-frequency-aware SfM, 3DGS self-initialization, and point-cloud regularization—to densify and stabilize seed points. Across LLFF and Mip-NeRF360, this initialization-first approach yields state-of-the-art sparse-view performance and complements existing regularizers such as DropGaussian.

Abstract

Sparse-view 3D Gaussian Splatting (3DGS) often overfits to the training views, leading to artifacts like blurring in novel view rendering. Prior work addresses it either by enhancing the initialization (\emph{i.e.}, the point cloud from Structure-from-Motion (SfM)) or by adding training-time constraints (regularization) to the 3DGS optimization. Yet our controlled ablations reveal that initialization is the decisive factor: it determines the attainable performance band in sparse-view 3DGS, while training-time constraints yield only modest within-band improvements at extra cost. Given initialization's primacy, we focus our design there. Although SfM performs poorly under sparse views due to its reliance on feature matching, it still provides reliable seed points. Thus, building on SfM, our effort aims to supplement the regions it fails to cover as comprehensively as possible. Specifically, we design: (i) frequency-aware SfM that improves low-texture coverage via low-frequency view augmentation and relaxed multi-view correspondences; (ii) 3DGS self-initialization that lifts photometric supervision into additional points, compensating SfM-sparse regions with learned Gaussian centers; and (iii) point-cloud regularization that enforces multi-view consistency and uniform spatial coverage through simple geometric/visibility priors, yielding a clean and reliable point cloud. Our experiments on LLFF and Mip-NeRF360 demonstrate consistent gains in sparse-view settings, establishing our approach as a stronger initialization strategy. Code is available at https://github.com/zss171999645/ItG-GS.

Figures (9)

• Figure 1: Ablation study on Mip-NeRF360 under the 12-view sparse-view protocol PSNR on average, comparing three training-time constraints (DropGaussian, CoR-GS, and FSGS) against vanilla 3DGS across different initialization strengths (Init_v6, Init_v12, Init_vall). Results show that while regularization methods modestly delay overfitting and improve performance within each initialization band, the final reconstruction quality is primarily governed by initialization strength.
• Figure 2: Our initialization pipeline for sparse-view 3DGS. Given sparse multi-view images with camera poses, we first mask high-frequency regions and perform SfM to obtain a raw point cloud, which better captures smooth areas. To compensate for regions lacking distinctive features, we train a lightweight first-pass 3DGS and incorporate its primitive centroids into the initialization. Finally, we apply clustering-based denoising, single-view filtering, and physics-based regularization to rectify the point cloud and suppress noise.
• Figure 3: Qualitative comparisons on Mip-NeRF360 dataset and LLFF dataset.
• Figure 4: Comparison of initialization quality and its impact on reconstruction performance. Our method provides more accurate initialization points, especially around scene edges, leading to improved final renderings.
• Figure 5: Performance curves between ours and baselines on the Mip-NeRF360 dataset.