POp-GS: Next Best View in 3D-Gaussian Splatting with P-Optimality

TL;DR

The paper tackles uncertainty quantification for 3D-Gaussian Splatting (3D-GS) by recasting information gain as a problem in optimal experimental design using P-Optimality. It derives a general covariance-based framework, then introduces scalable diagonal and block-diagonal Hessian approximations to enable practical Next-Best-View (NbV) selection, including single and batch view strategies. Across real and synthetic datasets, especially Mip-NeRF360 and Blender, D-Optimality and T-Optimality consistently outperform FisherRF and uniform baselines, with block-diagonal covariance offering additional gains in rendering fidelity (PSNR/SSIM/LPIPS) and better alignment with oracle guidance. The approach supports improved active perception and SLAM-like tasks with 3D-GS by providing principled view selection and a trade-off between computational cost and information fidelity. Overall, the work advances robust uncertainty-aware 3D-GS mapping and view planning with practical implications for robotics and immersive scene understanding.

Abstract

In this paper, we present a novel algorithm for quantifying uncertainty and information gained within 3D Gaussian Splatting (3D-GS) through P-Optimality. While 3D-GS has proven to be a useful world model with high-quality rasterizations, it does not natively quantify uncertainty or information, posing a challenge for real-world applications such as 3D-GS SLAM. We propose to quantify information gain in 3D-GS by reformulating the problem through the lens of optimal experimental design, which is a classical solution widely used in literature. By restructuring information quantification of 3D-GS through optimal experimental design, we arrive at multiple solutions, of which T-Optimality and D-Optimality perform the best quantitatively and qualitatively as measured on two popular datasets. Additionally, we propose a block diagonal covariance approximation which provides a measure of correlation at the expense of a greater computation cost.

Figures (6)

• Figure 1: We propose a novel method for calculating information gain from image in 3D Gaussian Splatting. In the above images, each method is provided a set of one hundred candidate views on scenes from the Blender dataset, and selects ten views to train a 3D-GS model on. Compared to state-of-the-art, our method more accurately estimate the information value of images to train a 3D-GS model. In this figure we demonstrate block D-Optimality, however our derivation also provides multiple solutions discussed later.
• Figure 2: The Hessian matrix captures the information content of each parameter in the trained 3D-GS model, approximated as per-pixel gradients over the training set of images. Since a trained 3D-GS model may contain millions of parameters, we approximate the Hessian matrix through a block or main diagonal.
• Figure 3: Optimal experimental design defines functionals of the eigenvalues of the covariance matrices, each with geometric intuitions. D-Optimality approximates the volume of the covariance matrix, as shown in this figure.
• Figure 4: Comparison of view selection methods on the Mip-Nerf360 dataset with 10 views. The columns are built by different methods and in order are: uniform sampling, FisherRF, Block D-GS, and the ground truth image.
• Figure 5: Correlation of expected information gain with PSNR of candidate views on two objects in Blender dataset.