VGNC: Reducing the Overfitting of Sparse-view 3DGS via Validation-guided Gaussian Number Control

TL;DR

VGNC addresses overfitting in sparse-view 3D Gaussian Splatting by introducing Validation-guided Gaussian Number Control, which leverages generative validation images to automatically determine the optimal Gaussian count during training. It generates validation views with a diffusion-based model, filters out distorted views via geometric consistency checks, and uses a validation monitor to guide growth and dropout of Gaussians, as well as joint initialization with validation images to improve initial geometry. Across five sparse-view 3DGS baselines and multiple datasets, VGNC reduces Gaussian redundancy, lowers memory and training/rendering time, and improves novel-view rendering quality. This approach demonstrates that carefully filtered, diffusion-generated validation data can effectively guide model complexity in 3D reconstruction, suggesting a practical path for incorporating validation-driven controls in sparse-view 3DGS systems.

Abstract

Sparse-view 3D reconstruction is a fundamental yet challenging task in practical 3D reconstruction applications. Recently, many methods based on the 3D Gaussian Splatting (3DGS) framework have been proposed to address sparse-view 3D reconstruction. Although these methods have made considerable advancements, they still show significant issues with overfitting. To reduce the overfitting, we introduce VGNC, a novel Validation-guided Gaussian Number Control (VGNC) approach based on generative novel view synthesis (NVS) models. To the best of our knowledge, this is the first attempt to alleviate the overfitting issue of sparse-view 3DGS with generative validation images. Specifically, we first introduce a validation image generation method based on a generative NVS model. We then propose a Gaussian number control strategy that utilizes generated validation images to determine the optimal Gaussian numbers, thereby reducing the issue of overfitting. We conducted detailed experiments on various sparse-view 3DGS baselines and datasets to evaluate the effectiveness of VGNC. Extensive experiments show that our approach not only reduces overfitting but also improves rendering quality on the test set while decreasing the number of Gaussian points. This reduction lowers storage demands and accelerates both training and rendering. The code will be released.

Figures (7)

• Figure 1: We evaluate the performance of five different methodsli2024dngaussianzhu2024fsgszhang2024corxiong2024sparsegskerbl20233d on the Mip-NeRF360 datasetbarron2022mip under sparse-view reconstruction, with and without our method. Integrating our approach improves rendering quality while reducing model storage across all methods.
• Figure 2: Overview of VGNC. We first propose a validation view generation method. Then, we introduce a validation-based monitor into the 3DGS training process to guide the control of the Gaussian quantity. This enables the model to automatically identify the optimal number of Gaussians during training.
• Figure 3: Gaussian number vs. other variables.
• Figure 4: Generated views with differing hallucinations.
• Figure 5: Qualitative results before and after integrating our approach.