These Magic Moments: Differentiable Uncertainty Quantification of Radiance Field Models

TL;DR

This paper tackles uncertainty quantification for radiance-field renderings by reframing the rendering equation as a probabilistic process and deriving differentiable, pixel-wise higher-order moments. By computing $\mathbb{E}[\rho]$, $\mathbb{E}[\rho^2]$, and thus $\mathrm{Var}[\rho] = \mathbb{E}[\rho^2]-\mathbb{E}[\rho]^2$, the method yields fast, end-to-end uncertainty signals for color, depth, and semantics in both NeRF and 3D Gaussian Splatting (3DGS). It demonstrates superior or comparable performance to state-of-the-art uncertainty methods while offering significant runtime advantages, and it shows practical utility in next-best-view planning and active ray sampling for neural radiance-field training. The proposed framework enables differentiable, uncertainty-aware decision making in view planning and data acquisition for radiance-field representations.

Abstract

This paper introduces a novel approach to uncertainty quantification for radiance fields by leveraging higher-order moments of the rendering equation. Uncertainty quantification is crucial for downstream tasks including view planning and scene understanding, where safety and robustness are paramount. However, the high dimensionality and complexity of radiance fields pose significant challenges for uncertainty quantification, limiting the use of these uncertainty quantification methods in high-speed decision-making. We demonstrate that the probabilistic nature of the rendering process enables efficient and differentiable computation of higher-order moments for radiance field outputs, including color, depth, and semantic predictions. Our method outperforms existing radiance field uncertainty estimation techniques while offering a more direct, computationally efficient, and differentiable formulation without the need for post-processing. Beyond uncertainty quantification, we also illustrate the utility of our approach in downstream applications such as next-best-view (NBV) selection and active ray sampling for neural radiance field training. Extensive experiments on synthetic and real-world scenes confirm the efficacy of our approach, which achieves state-of-the-art performance while maintaining simplicity.

Figures (12)

• Figure 1: Our proposed method quantifies uncertainty over rendered scenes generated through radiance field models by considering the differentiable higher-order moments of the rendering equation. The proposed approach requires no additional training, generalizes for color, depth, and semantics, and may be used for both NeRF and 3DGS methods. Shown is the variance, or second central moment, which is highly correlated to the rendering error for both color and semantics.
• Figure 2: Images containing color, depth, semantics, or other features of interest, along with camera poses and an initial point cloud, are used to train a radiance field model depicted. The rendered features (i.e. the expected feature of each pixel) are computed in a differentiable manner. Our proposed approach demonstrates that the same probabilistic process used to compute these renders can be used to differentiably compute the higher-order moments. Solid arrows indicate operation flow and dotted arrows indicate gradient flow.
• Figure 3: Uncertainty quantification for the depth estimates of a NeRF and 3DGS model. A NeRF model is trained and we compare the Bayes' Rays goli2024bayes uncertainty estimate with our proposed variance estimate. Similarly, a 3DGS model is trained and we compare the FisherRF jiang2023fisherrf uncertainty estimate with our proposed variance estimate. Unlike existing methods, our approach works with both NeRF and 3DGS models. The CF-NeRF yan2024cf and 3DGS Ensemble sunderhauf2023density methods are not shown as they require specialized models for uncertainty quantification.
• Figure 4: Rendering quality using active ray selection for the ficus scene in the Blender dataset. For the NeRF baseline, rays are selected uniformly over the entire image. iMAP sucar2021imap and the proposed approach sample for high error and high variance regions of the image, respectfully. The proposed approach matches the performance of iMAP without needing to reference the ground-truth image.
• Figure 5: Given a radiance field model that is trained using only a small number views, the goal is to find the next best view, or the view that improves the radiance field model the most. The rendered variance four four views is shown. The highlighted viewpoint has the highest pixel-wise variance and is therefore chosen as the NBV.