Manifold Sampling for Differentiable Uncertainty in Radiance Fields

TL;DR

This work proposes a versatile approach for learning Gaussian radiance fields with explicit and fine-grained uncertainty estimates that impose only little additional cost compared to uncertainty-agnostic training, and demonstrates state-of-the-art performance on next-best-view planning tasks.

Abstract

Radiance fields are powerful and, hence, popular models for representing the appearance of complex scenes. Yet, constructing them based on image observations gives rise to ambiguities and uncertainties. We propose a versatile approach for learning Gaussian radiance fields with explicit and fine-grained uncertainty estimates that impose only little additional cost compared to uncertainty-agnostic training. Our key observation is that uncertainties can be modeled as a low-dimensional manifold in the space of radiance field parameters that is highly amenable to Monte Carlo sampling. Importantly, our uncertainties are differentiable and, thus, allow for gradient-based optimization of subsequent captures that optimally reduce ambiguities. We demonstrate state-of-the-art performance on next-best-view planning tasks, including high-dimensional illumination planning for optimal radiance field relighting quality.

Figures (7)

• Figure 1: Using the same four training views (top), different model parameter initializations yield different radiance fields (bottom), reflecting uncertainty.
• Figure 2: Different variants to model an uncertainty volume $V$V in the space of radiance field parameters (top row, only three out of millions of parameters are shown) using different covariance matrices $\Sigma$Σ (bottom row, 20 dimensions are shown). (a) A full $\Sigma$Σ is the most expressive solution that leads to an arbitrarily shaped parallelotope, but it suffers from an intractable number of parameters. (b) Restricting $\Sigma$Σ to a diagonal matrix is a sparse solution, but it can only represent axis-aligned hyper-rectangles. (c) A block-diagonal $\Sigma$Σ is slightly more expressive, but it requires making representation-specific independence assumptions and small blocks to stay tractable. (d) Our solution employs a low-rank covariance matrix, which results in a manifold parallelotope (here a 2D parallelogram). This parameterization is highly efficient to train and results in expressive uncertainty estimates.
• Figure 3: Active camera planning results on the NeRF Synthetic dataset from different methods (columns) using 10 training views. The first row of each scene shows a novel view, while the second row illustrates the distribution of cameras placed by each method. Red cones represent the initial camera position, green cones represent the first four camera positions, and blue cones represent the final five camera positions.
• Figure 4: Active camera planning results on the MipNeRF-360 dataset from different methods (columns) using 20 training views. The images show a novel view.
• Figure 5: Active illumination planning results. We demonstrate novel-view relighting results based on 8 training illumination conditions. We display two test illuminations per scene, shown as insets in the respective reference image.