CoR-GS: Sparse-View 3D Gaussian Splatting via Co-Regularization

TL;DR

CoR-GS tackles the overfitting problem of sparse-view 3D Gaussian Splatting by introducing a co-regularization framework that trains two independent Gaussian radiance fields. It defines two high-signal disagreement metrics—point disagreement on Gaussian positions and rendering disagreement on pixel outputs—and shows their disagreements negatively correlate with reconstruction quality, enabling unsupervised detection of inaccuracies. The method combines co-pruning, which removes Gaussians with poor cross-field correspondences, and pseudo-view co-regularization, which enforces color consistency across pseudo views, yielding coherent, compact geometry and state-of-the-art sparse-view synthesis across LLFF, Mip-NeRF360, DTU, and Blender. This approach improves reliability and efficiency of 3DGS under limited views and offers a generalizable framework for regularizing unstructured radiance fields without external supervision.

Abstract

3D Gaussian Splatting (3DGS) creates a radiance field consisting of 3D Gaussians to represent a scene. With sparse training views, 3DGS easily suffers from overfitting, negatively impacting rendering. This paper introduces a new co-regularization perspective for improving sparse-view 3DGS. When training two 3D Gaussian radiance fields, we observe that the two radiance fields exhibit point disagreement and rendering disagreement that can unsupervisedly predict reconstruction quality, stemming from the randomness of densification implementation. We further quantify the two disagreements and demonstrate the negative correlation between them and accurate reconstruction, which allows us to identify inaccurate reconstruction without accessing ground-truth information. Based on the study, we propose CoR-GS, which identifies and suppresses inaccurate reconstruction based on the two disagreements: (1) Co-pruning considers Gaussians that exhibit high point disagreement in inaccurate positions and prunes them. (2) Pseudo-view co-regularization considers pixels that exhibit high rendering disagreement are inaccurate and suppress the disagreement. Results on LLFF, Mip-NeRF360, DTU, and Blender demonstrate that CoR-GS effectively regularizes the scene geometry, reconstructs the compact representations, and achieves state-of-the-art novel view synthesis quality under sparse training views.

Figures (20)

• Figure 1: Illustration of how the different behaviors between two 3D Gaussian radiance fields correlated to construction quality. Gaussians with different behaviors tend to not fit the ground-truth shape well. Therefore, inaccurate reconstructions can be identified by measuring the differences without accessing ground-truth information.
• Figure 2: The recorded different behaviors of two 3d Gaussian radiance fields during training. The point disagreement and rendering disagreement increases during training, especially during densification.
• Figure 3: The recorded different behaviors of two 3d Gaussian radiance fields during training. The point disagreement and rendering disagreement increases during training, especially during densification.
• Figure 4: The correlation between the two disagreements and reconstruction quality. The x-axis percentage represents we mask out the percentage of regions with the highest disagreement scores. With the reduction of regions with higher disagreement scores, the reconstruction quality averaging the remaining regions continuously improves.
• Figure 5: Test-view visualization of rendered images and depth maps, and the corresponding error maps. The disagreed regions between the rendered results of two 3D Gaussian radiance fields tend to be inaccurate compared to ground truth.