Instant Uncertainty Calibration of NeRFs Using a Meta-Calibrator

TL;DR

A meta-calibrator is a neural network that takes as input the NeRF images and uncalibrated uncertainty maps and outputs a scene-specific calibration curve that corrects the NeRF's uncalibrated uncertainties, significantly beating DANE and other approaches.

Abstract

Although Neural Radiance Fields (NeRFs) have markedly improved novel view synthesis, accurate uncertainty quantification in their image predictions remains an open problem. The prevailing methods for estimating uncertainty, including the state-of-the-art Density-aware NeRF Ensembles (DANE) [29], quantify uncertainty without calibration. This frequently leads to over- or under-confidence in image predictions, which can undermine their real-world applications. In this paper, we propose a method which, for the first time, achieves calibrated uncertainties for NeRFs. To accomplish this, we overcome a significant challenge in adapting existing calibration techniques to NeRFs: a need to hold out ground truth images from the target scene, reducing the number of images left to train the NeRF. This issue is particularly problematic in sparse-view settings, where we can operate with as few as three images. To address this, we introduce the concept of a meta-calibrator that performs uncertainty calibration for NeRFs with a single forward pass without the need for holding out any images from the target scene. Our meta-calibrator is a neural network that takes as input the NeRF images and uncalibrated uncertainty maps and outputs a scene-specific calibration curve that corrects the NeRF's uncalibrated uncertainties. We show that the meta-calibrator can generalize on unseen scenes and achieves well-calibrated and state-of-the-art uncertainty for NeRFs, significantly beating DANE and other approaches. This opens opportunities to improve applications that rely on accurate NeRF uncertainty estimates such as next-best view planning and potentially more trustworthy image reconstruction for medical diagnosis. The code is available at https://niki-amini-naieni.github.io/instantcalibration.github.io/.

Figures (12)

• Figure 1: We propose a method for efficiently calibrating the uncertainties from NeRF models. Our approach is based on a meta-calibrator that takes as input features from the rendered NeRF images and uncalibrated uncertainty maps and predicts the calibration function, $R_{\boldsymbol{\theta}}(\cdot)$, for the NeRF model. Our meta-calibrator generalizes across scenes so it only needs to be trained once, and can predict the calibration function in a single forward pass without any ground truth data from the target scene.
• Figure 2: Meta-calibrator design. In stage (a) we fit a low-dimensional parameteric model of the calibration curves. The meta-calibrator then predicts these curve parameters from rendered images of the scene and their associated uncalibrated uncertainty maps (b).
• Figure 3: Meta-calibrator design decisions. Results showing using 3 components for Principal Component Analysis (PCA) model of calibration curves, 21 scenes to fit PCA model, and 30 scenes to train meta-calibrator achieves good generalization to new test scenes.
• Figure 4: Quantitative comparison of uncalibrated and calibrated uncertainties. In (a), we show calibration curves on test data from four scenes in LLFF llff. The color of each curve indicates the color channel it corresponds to. The calibrated curves are much closer to the ideal calibration (dashed lines), demonstrating that the meta-calibrator works very well. In (b), the average calibration error and negative log-likelihood before and after calibration are reported for LLFF, clearly showing the meta-calibrator improves the accuracy of the uncertainties (lowering calibration error and negative log-likelihood). To test generalization, the meta-calibrator was also applied to held-out scenes in DTU dtu, achieving a 70 % reduction in calibration error on average.
• Figure 5: Comparison to DANE da-nerf. Results comparing our uncertainties to those from the state-of-the-art method DANE. In (a) we show DANE's RGB calibration curves are not closely aligned with the perfectly calibrated lines, meaning it is miscalibrated. It is significantly over-confident for expected confidence levels close to 1 and under-confident for confidence levels close to 0. In comparison, the curves for our approach are extremely close to the ideal calibration (dashed lines), demonstrating that the meta-calibrator works very well, predicting expected confidences that match the true ones. This is also verified by how our calibration error is over two orders of magnitude smaller than DANE's. In (b) we show that our approach results in more efficient performance gains over DANE for next-best view planning.