Density Uncertainty Quantification with NeRF-Ensembles: Impact of Data and Scene Constraints

TL;DR

This work tackles the problem that density predictions in NeRFs critically determine 3D geometry and surface reconstructions. It introduces NeRF-Ensembles to quantify per-voxel density uncertainty by training multiple independent NeRFs and discretizing the scene into a 3D grid to compute mean density and uncertainty, enabling posterior insights into reconstruction reliability. The study shows that data constraints (image and pose noise) and scene constraints (acquisition constellation, occlusions, material properties) elevate density uncertainty and degrade training, yet ensemble averaging improves robustness and enables uncertainty-driven artifact removal via percentile thresholds. The findings offer a practical framework for uncertainty-aware NeRF reconstructions across synthetic and real datasets, with tangible benefits for post-processing and improved 3D scene completeness.

Abstract

In the fields of computer graphics, computer vision and photogrammetry, Neural Radiance Fields (NeRFs) are a major topic driving current research and development. However, the quality of NeRF-generated 3D scene reconstructions and subsequent surface reconstructions, heavily relies on the network output, particularly the density. Regarding this critical aspect, we propose to utilize NeRF-Ensembles that provide a density uncertainty estimate alongside the mean density. We demonstrate that data constraints such as low-quality images and poses lead to a degradation of the training process, increased density uncertainty and decreased predicted density. Even with high-quality input data, the density uncertainty varies based on scene constraints such as acquisition constellations, occlusions and material properties. NeRF-Ensembles not only provide a tool for quantifying the uncertainty but exhibit two promising advantages: Enhanced robustness and artifact removal. Through the utilization of NeRF-Ensembles instead of single NeRFs, small outliers are removed, yielding a smoother output with improved completeness of structures. Furthermore, applying percentile-based thresholds on density uncertainty outliers proves to be effective for the removal of large (foggy) artifacts in post-processing. We conduct our methodology on 3 different datasets: (i) synthetic benchmark dataset, (ii) real benchmark dataset, (iii) real data under realistic recording conditions and sensors.

Figures (12)

• Figure 1: Flowchart of the methodology: Input data are the images and corresponding camera poses. A NeRF-Ensemble with a total number of M members is trained. The NeRF-Ensemble provides an average network output with mean density values over all NeRFs in the ensemble and a corresponding density uncertainty quantification of the prediction in 3D space. Optionally (shown in gray) we add noise to the input data and investigate the effect of data constraints on the results. In addition, a subsequent removing of high uncertainty points, e.g. with percentiles, offers the opportunity to remove (foggy) artifacts.
• Figure 2: Histograms displaying the frequency of the density uncertainty $\text{U}_{\delta}$ of all points in the 3D grid for the NeRF synthetic dataset. Shown are the scenes under different input data types: Original, image noise $\sigma_{\text{Im}}$, pose noise $\sigma_{\text{t}}$ (translation), pose noise $\sigma_{\text{R}}$ (rotation) and pose noise $\sigma_{\text{t,R}}$ (translation + rotation).
• Figure 3: Histograms displaying the frequency of the density uncertainty $\text{U}_{\delta}$ of all points in the 3D grid for the DTU dataset. Shown are the scenes under different input data types: Original, image noise $\sigma_{\text{Im}}$, pose noise $\sigma_{\text{t}}$ (translation), pose noise $\sigma_{\text{R}}$ (rotation) and pose noise $\sigma_{\text{t,R}}$ (translation + rotation).
• Figure 4: Spider chart showing the density uncertainty for NeRF synthetic (left) and DTU dataset (right). Displayed are the mean density uncertainty $\text{mU}_{\delta}$ in the 3D grid for different types of input data: original, image noise $\sigma_{\text{Im}}$, pose noise (translation) $\sigma_{\text{t}}$, pose noise (rotation) $\sigma_{\text{R}}$ and pose noise (translation + rotation) $\sigma_{\text{R,t}}$.
• Figure 5: Qualitative comparison on the NeRF synthetic dataset, showing points in the 3D grid with density above 15. Shown are the individual scenes for single NeRF (column 1), NeRF-Ensemble (column 2) as well as the density uncertainty $\text{U}_{\delta}$ under different input data configurations: Original (column 3), image noise $\sigma_{\text{Im}}$ (column 4), pose noise $\sigma_{\text{t}}$ (column 5) and pose noise $\sigma_{\text{R}}$ (column 6). Density uncertainty values above 3 are set to 3 for clearer visualization.