SA-ResGS: Self-Augmented Residual 3D Gaussian Splatting for Next Best View Selection

TL;DR

SA-ResGS addresses unstable uncertainty and under-supervised 3D Gaussian Splatting during NBV-driven active reconstruction by introducing Self-Augmented Points for physically grounded view selection and a residual supervision scheme that strengthens gradient flow to weak Gaussians. The framework couples a hash-based surface-coverage representation with uncertainty-aware training to decouple early NBV decisions from volatile learning dynamics, yielding more uniform coverage and better-calibrated uncertainty. Empirical results across NeRF-Synthetic, Mip-NeRF 360, and extended NBV datasets show improved reconstruction quality and robustness, along with substantial speedups in view-selection computation. This approach offers a scalable, uncertainty-aware NBV paradigm with practical benefits for real-world active scene capture and 3D reconstruction.

Abstract

We propose Self-Augmented Residual 3D Gaussian Splatting (SA-ResGS), a novel framework to stabilize uncertainty quantification and enhancing uncertainty-aware supervision in next-best-view (NBV) selection for active scene reconstruction. SA-ResGS improves both the reliability of uncertainty estimates and their effectiveness for supervision by generating Self-Augmented point clouds (SA-Points) via triangulation between a training view and a rasterized extrapolated view, enabling efficient scene coverage estimation. While improving scene coverage through physically guided view selection, SA-ResGS also addresses the challenge of under-supervised Gaussians, exacerbated by sparse and wide-baseline views, by introducing the first residual learning strategy tailored for 3D Gaussian Splatting. This targeted supervision enhances gradient flow in high-uncertainty Gaussians by combining uncertainty-driven filtering with dropout- and hard-negative-mining-inspired sampling. Our contributions are threefold: (1) a physically grounded view selection strategy that promotes efficient and uniform scene coverage; (2) an uncertainty-aware residual supervision scheme that amplifies learning signals for weakly contributing Gaussians, improving training stability and uncertainty estimation across scenes with diverse camera distributions; (3) an implicit unbiasing of uncertainty quantification as a consequence of constrained view selection and residual supervision, which together mitigate conflicting effects of wide-baseline exploration and sparse-view ambiguity in NBV planning. Experiments on active view selection demonstrate that SA-ResGS outperforms state-of-the-art baselines in both reconstruction quality and view selection robustness.

Figures (16)

• Figure 1: Overview of SA-ResGS. The framework alternates between view selection and training. At each NBV step, Self-Augmented Points are generated via triangulation from dense correspondences between a training view and its extrapolated render, enabling surface-aware coverage estimation (Sec.\ref{['subsec:4_1']}). Candidate views are first physically filtered using hash-encoded feature dissimilarity, then ranked by uncertainty quantification scores for final selection (Sec.\ref{['subsec:4_2']}). During training, residual supervision (Sec. \ref{['subsec:4_4']}) combines full and uncertainty-intensified renders to reinforce gradients toward weakly contributing Gaussians, improving training stability and reconstruction quality under sparse-view conditions.
• Figure 2: SA-Points Generation. An extrapolated image is rendered from a perturbed camera pose. Dense correspondences with the reference image are predicted using MASt3R, and triangulated to produce SA-Points, which are filtered by reprojection error for reliable surface geometry.
• Figure 3: Physically grounded candidate view selection via surface coverage. (a) SA-Points from training views define observed voxels $\mathcal{V}_{\text{obs}}$. (b) Each candidate view generates a binary hash-encoded feature $\mathbf{b}$, via frustum-based visibility estimation. (c) Normalized Hamming distance between hash-encoded features quantifies coverage dissimilarity, enabling efficient selection of geometrically complementary views without rendered images or uncertainty scores.
• Figure 4: Residual supervision in 3DGS. (a) At each iteration ($t_1$, $t_2$),$\mathcal{G}_{\text{sup}}$ combines random and top-uncertain Gaussians; (b) residual supervision in 3DGS mimics ResNet-style skip connections.
• Figure 5: Qualitative Comparison of Active View Selection. Reconstruction from 20 selected views per scene. Our method shows improved completeness and fewer artifacts compared to baselines. For multi-view visualization, please refer to the Appendix. Sec. \ref{['Supp:Additional_qual']} and supplementary video.