Peering into the Unknown: Active View Selection with Neural Uncertainty Maps for 3D Reconstruction

TL;DR

This work introduces a novel AVS approach guided by neural uncertainty maps predicted by a lightweight feedforward deep neural network, named UPNet, which significantly reduces computational overhead during AVS, achieving up to a 400 times speedup along with over 50\% reductions in CPU, RAM, and GPU usage compared to baseline methods.

Abstract

Some perspectives naturally provide more information than others. How can an AI system determine which viewpoint offers the most valuable insight for accurate and efficient 3D object reconstruction? Active view selection (AVS) for 3D reconstruction remains a fundamental challenge in computer vision. The aim is to identify the minimal set of views that yields the most accurate 3D reconstruction. Instead of learning radiance fields, like NeRF or 3D Gaussian Splatting, from a current observation and computing uncertainty for each candidate viewpoint, we introduce a novel AVS approach guided by neural uncertainty maps predicted by a lightweight feedforward deep neural network, named UPNet. UPNet takes a single input image of a 3D object and outputs a predicted uncertainty map, representing uncertainty values across all possible candidate viewpoints. By leveraging heuristics derived from observing many natural objects and their associated uncertainty patterns, we train UPNet to learn a direct mapping from viewpoint appearance to uncertainty in the underlying volumetric representations. Next, our approach aggregates all previously predicted neural uncertainty maps to suppress redundant candidate viewpoints and effectively select the most informative one. Using these selected viewpoints, we train 3D neural rendering models and evaluate the quality of novel view synthesis against other competitive AVS methods. Remarkably, despite using half of the viewpoints than the upper bound, our method achieves comparable reconstruction accuracy. In addition, it significantly reduces computational overhead during AVS, achieving up to a 400 times speedup along with over 50\% reductions in CPU, RAM, and GPU usage compared to baseline methods. Notably, our approach generalizes effectively to AVS tasks involving novel object categories, without requiring any additional training.

Figures (9)

• Figure 1: (a) Illustration of Active Viewpoint Selection (AVS). The goal of AVS is to select the most informative viewpoints (red cones) from candidate views (gray cones) to minimize reconstruction error between the ground-truth view and the novel view synthesized by rendering methods such as NeRF mildenhall2021nerf or 3DGS kerbl20233d. (b) Neural Uncertainty Map (UMap). For each selected viewpoint, our method predicts a UMap in polar coordinates, assigning uncertainty values to candidate viewpoints on a spherical surface. A viewpoint is parameterized by azimuth $\phi \in [0^\circ,360^\circ]$ (red) and elevation $\theta \in [0^\circ,180^\circ]$ (blue), at a fixed radius from the current view. The coordinate system is oriented with the z-axis pointing from the object center to the current view, the y-axis upward, and the x-axis to the right. A sample viewpoint is shown as a black dot, with uncertainty visualized by the colorbar. See Sec. \ref{['sec:umap_dataset']} for details.
• Figure 2: (a) Pipeline for Creating the Neural Uncertainty Map (NUM) Dataset. Given a current viewpoint of a 3D object, its fixed anchor viewpoints ${p_j}$ (red dots) are defined around a 3D object, with ground-truth images rendered in Blender. A single-view view synthesis method, SplatterImage szymanowicz2024splatter synthesizes novel views at these anchors, and reconstruction errors (PSNR, SSIM, LPIPS, or MSE) are projected into polar coordinates to form the UMaps. (b) Overview of our proposed Peering into the Unknown (PUN) method for AVS. From the current view $v_t$, our transformer-based UPNet takes image patches as input and predicts an uncertainty map $U_t$. PUN integrates past UMaps ($t=0$ to $t$, red triangles), filters redundant viewpoints, and selects the next best view $v_{t+1}$ (blue triangles). This process repeats iteratively to guide exploration.
• Figure 3: Visualization of ground-truth and predicted uncertainty maps by our PUN method. We present two examples from: (a) NUM-inst and (b) NUM-cat. In each case, the input view $I_t$ at the current viewpoint $v_t$ is fed into UPNet to produce the predicted UMap. The corresponding ground-truth UMap is shown alongside for comparison. Both maps are min-max normalized to $[0,1]$ for visualization only. See the colorbar for uncertainty values and polar coordinates for the UMaps.
• Figure 4: Visualization of the datasets used for AVS evaluation. From left to right, the object instances exhibit increasing geometric complexity, occlusion, and background clutter.
• Figure S1: The data distribution of our NUM dataset. In total, we select 13 categories from the ShapeNet dataset chang2015shapenet, with 100 object instances per category used to generate our NUM dataset. The blue regions indicate the training dataset for our UPNet, consisting of 90 instances from each of 11 categories. The yellow regions represent the instances in NUM-inst. The green regions show the instances in NUM-cat.