GauSS-MI: Gaussian Splatting Shannon Mutual Information for Active 3D Reconstruction

TL;DR

A probabilistic model that quantifies visual uncertainty for each Gaussian, Gaussian Splatting Shannon Mutual Information (GauSS-MI), is introduced, for real-time assessment of visual mutual information from novel viewpoints, facilitating the selection of next best view.

Abstract

This research tackles the challenge of real-time active view selection and uncertainty quantification on visual quality for active 3D reconstruction. Visual quality is a critical aspect of 3D reconstruction. Recent advancements such as Neural Radiance Fields (NeRF) and 3D Gaussian Splatting (3DGS) have notably enhanced the image rendering quality of reconstruction models. Nonetheless, the efficient and effective acquisition of input images for reconstruction-specifically, the selection of the most informative viewpoint-remains an open challenge, which is crucial for active reconstruction. Existing studies have primarily focused on evaluating geometric completeness and exploring unobserved or unknown regions, without direct evaluation of the visual uncertainty within the reconstruction model. To address this gap, this paper introduces a probabilistic model that quantifies visual uncertainty for each Gaussian. Leveraging Shannon Mutual Information, we formulate a criterion, Gaussian Splatting Shannon Mutual Information (GauSS-MI), for real-time assessment of visual mutual information from novel viewpoints, facilitating the selection of next best view. GauSS-MI is implemented within an active reconstruction system integrated with a view and motion planner. Extensive experiments across various simulated and real-world scenes showcase the superior visual quality and reconstruction efficiency performance of the proposed system.

Figures (10)

• Figure 1: Illustration of the proposed Gaussian Splatting Shannon Mutual Information (GauSS-MI) method. Upper part: At each active reconstruction step, once a new observation is obtained, the 3D Gaussian Splatting (3DGS) map is updated and optimized by minimizing the image loss between observed images and the map. To quantify visual uncertainty, we construct a probabilistic model for each 3D Gaussian ellipsoid by mapping residual image loss onto the 3DGS map. Using this model, we define GauSS-MI, a metric that estimates mutual information between the reconstruction model and a viewpoint. GauSS-MI enables real-time visual quality assessment from novel viewpoints without a prior, facilitating the selection of the next-best-view to effectively reduce map uncertainty. Lower part: The active reconstruction process iterates and decreases visual uncertainty, resulting in a high visual fidelity 3D reconstruction result.
• Figure 2: Inverse sensor model visualization. The hyperparameters $\lambda$ are omitted in the figure for simplicity.
• Figure 3: Overview of proposed active 3D reconstruction system.
• Figure 4: High-resolution novel view synthesis of the reconstruction result by the proposed system: color rendering against depth rendering.
• Figure 5: PSNR results for active view selection with a limited number of frames. The maximum PSNR value for each test is annotated. The abbreviations 'G' and 'F' denote GauSS-MI and FisherRF, respectively.