Surface-Based Visibility-Guided Uncertainty for Continuous Active 3D Neural Reconstruction

TL;DR

This work introduces Surface-Based Visibility Field (SBV) to estimate visibility-guided uncertainty during continuous active 3D neural reconstruction. By coupling neural implicit surfaces (NeuS) with a voxel-grid that tracks surface confidences, SBV computes a surface-aware information gain that drives multiple next-best views even in underfitted training stages. The approach yields robust improvements across diverse benchmarks (DTU, Blender, TanksAndTemples, BlendedMVS) and a new imbalanced-view dataset (ImBView), demonstrating heightened resilience to occlusions and incomplete surfaces. The results show up to 11.6% gains in image rendering quality and improved mesh reconstruction, highlighting SBV's practical impact for efficient, data-aware 3D reconstruction.

Abstract

View selection is critical in active 3D neural reconstruction as it impacts the contents of training set and resulting final output quality. Recent view selection strategies emphasize the visibility when evaluating model uncertainty in active 3D reconstruction. However, existing approaches estimate visibility only after the model fully converges, which has confined their application primarily to non-continuous active learning settings. This paper proposes Surface-Based Visibility field (SBV) that successfully estimates the visibility-guided uncertainty in continuous active 3D neural reconstruction. During learning neural implicit surfaces, our model learns rendering uncertainties and infers surface confidence values derived from signed distance functions. It then updates surface confidences using a voxel grid, robustly deducing the surface-based visibility for uncertainties. This approach captures uncertainties across all regions, whether well-defined surfaces or ambiguous areas, ensuring accurate visibility measurement in continuous active learning. Experiments on benchmark datasets-Tanks and Temples, BlendedMVS, Blender, DTU-and the newly proposed imbalanced viewpoint dataset (ImBView) show that view selection based on SBV-guided uncertainty improves performance by up to 11.6% over existing methods, highlighting its effectiveness in challenging reconstruction scenarios.

Figures (19)

• Figure 1: Problem statement. In continuous active 3D neural reconstruction, a model is initialized with a small number of views (blue) and evaluates the information gain of a candidate view (green) for next-best view selection. Regions trained with low data (b) exhibit lower convergence than those trained with high data (a). Prior methods often fail to capture uncertainties in underfitted regions with low volume densities, while our method performs well in both regions.
• Figure 2: Analysis of density-based visibility in continuous active 3D neural reconstruction. Volume density, visualized based on bitfield values, grows from low (yellow) to high (black) as additional views are included and training progresses, while uncertainty decreases from high to low. Gray regions represent unobservable areas from limited training views in the early stages of training.
• Figure 3: Scenarios for uncertainty estimation using surface-based visibility. Voxels containing inferred surfaces are highlighted in red, while voxels marked with diagonal red lines indicate low surface confidence in the current training stage but potentially form part of surface. (a): Rays traversing no visible surfaces (rays (1) and (4)) versus visible surfaces (rays (2) and (3)) exhibit distinct surface-based visibilities. (b): Demonstration of inferring surface confidence using estimated SDF. 3D points (a), (b), and (c) are points on same camera ray. Distance between (a) and (c) is sampling step size $1/s$ inferred from neural implicit surface network, with (b) being the midpoint.
• Figure 4: Visualization of surface-based visibility-guided uncertainty as training progresses. IG quantified with SBV is normalized using two methods: intra-image (I.N.) and global normalization (G.N.). Former normalizes image using its minimum and maximum IG values. In contrast, latter normalizes images using minimum of IG from 60K iterations and maximum of IG from 1K iterations.
• Figure 5: Types of viewpoints in the ImBView dataset. Distribution of view types: common view (75% of the training set, 40% of the test set), high-angle view (12.5% of the training set, 40% of the test set), low-angle view (12.5% of the training set, 20% of the test set).