UC-NeRF: Uncertainty-aware Conditional Neural Radiance Fields from Endoscopic Sparse Views

TL;DR

This paper proposes uncertainty-aware conditional NeRF for novel view synthesis to tackle the severe shape-radiance ambiguity from sparse surgical views, and incorporates the multi-view uncertainty estimation to condition the neural radiance field for modeling the severe photometric inconsistencies adaptively.

Abstract

Visualizing surgical scenes is crucial for revealing internal anatomical structures during minimally invasive procedures. Novel View Synthesis is a vital technique that offers geometry and appearance reconstruction, enhancing understanding, planning, and decision-making in surgical scenes. Despite the impressive achievements of Neural Radiance Field (NeRF), its direct application to surgical scenes produces unsatisfying results due to two challenges: endoscopic sparse views and significant photometric inconsistencies. In this paper, we propose uncertainty-aware conditional NeRF for novel view synthesis to tackle the severe shape-radiance ambiguity from sparse surgical views. The core of UC-NeRF is to incorporate the multi-view uncertainty estimation to condition the neural radiance field for modeling the severe photometric inconsistencies adaptively. Specifically, our UC-NeRF first builds a consistency learner in the form of multi-view stereo network, to establish the geometric correspondence from sparse views and generate uncertainty estimation and feature priors. In neural rendering, we design a base-adaptive NeRF network to exploit the uncertainty estimation for explicitly handling the photometric inconsistencies. Furthermore, an uncertainty-guided geometry distillation is employed to enhance geometry learning. Experiments on the SCARED and Hamlyn datasets demonstrate our superior performance in rendering appearance and geometry, consistently outperforming the current state-of-the-art approaches. Our code will be released at https://github.com/wrld/UC-NeRF.

Figures (9)

• Figure 1: Employing NeRF mildenhall2021nerf in surgical scene encounters two main challenges, i.e. endoscopic sparse views and photometric inconsistency.
• Figure 2: Training NeRF on sparse surgical views is challenging. NeRF mildenhall2021nerf fails to produce desirable views given sparse surgical scenes as inputs. State-of-the-art few-shot NeRF methods MonoSDF yu2022monosdf and GeoNeRF johari2022geonerf show degeneration in geometry rendering results. In contrast, our approach presents consistent improvement and achieves faster convergence in 4k compared to the 50k optimization of other baselines.
• Figure 3: Overview of our Uncertainty-aware Conditional NeRF (UC-NeRF). We first build a consistency learner upon the multi-view stereo network, to capture the view-consistent constraints to generate the uncertainty map. Then, the uncertainty-aware NeRF takes image features (from FPN and 3D regularization module) and the uncertainty map as input to predict the radiance field, resulting in reduced shape-radiance ambiguity and improved rendering accuracy. Finally, we introduce the distillation from geometry priors for further optimizing the neural rendering results.
• Figure 4: Visualization of the Sparse SfM depth and the estimated uncertainty map. With the guidance from SfM, the uncertainty map measures the extent of photometric inconsistency. The masked target view indicates the region with uncertainty larger than the mean value.
• Figure 5: Qualitative Comparisons of rendered color and depth. Given sparse input views, existing approaches show rendering results with blur and artifacts, suffering from photometric inconsistency. Our UC-NeRF can generate fine-grained details and consistent depth.