NeRF in detail: Learning to sample for view synthesis

TL;DR

This work replaces NeRF's handcrafted hierarchical sampling with a differentiable proposer that learns where to sample along rays, enabling end-to-end optimization. It evaluates multiple lightweight proposer architectures and introduces a two-stage training strategy to stabilize learning, achieving state-of-the-art results on Blender and competitive performance on LLFF-NeRF. Additionally, the model can predict sample importance to prune computations, yielding about 25% faster rendering without substantial quality loss. The proposed NeRF-ID framework is compatible with existing NeRF extensions and offers a practical path to more accurate and efficient view synthesis.

Abstract

Neural radiance fields (NeRF) methods have demonstrated impressive novel view synthesis performance. The core approach is to render individual rays by querying a neural network at points sampled along the ray to obtain the density and colour of the sampled points, and integrating this information using the rendering equation. Since dense sampling is computationally prohibitive, a common solution is to perform coarse-to-fine sampling. In this work we address a clear limitation of the vanilla coarse-to-fine approach -- that it is based on a heuristic and not trained end-to-end for the task at hand. We introduce a differentiable module that learns to propose samples and their importance for the fine network, and consider and compare multiple alternatives for its neural architecture. Training the proposal module from scratch can be unstable due to lack of supervision, so an effective pre-training strategy is also put forward. The approach, named `NeRF in detail' (NeRF-ID), achieves superior view synthesis quality over NeRF and the state-of-the-art on the synthetic Blender benchmark and on par or better performance on the real LLFF-NeRF scenes. Furthermore, by leveraging the predicted sample importance, a 25% saving in computation can be achieved without significantly sacrificing the rendering quality.

Figures (13)

• Figure 1: Overview of NeRF and our method. NeRF's coarse-to-fine approach (a) relies on a heuristic 'proposer' (b) which acts on the output of the coarse network and produces samples to pass to the fine network. We substitute this mechanism with a learnable proposer (c).
• Figure 2: Trainable proposer architectures. The input features come from the course network, as shown in Figures \ref{['fig:nerfgen']}(a) and (c). Full details are in Appendix \ref{['sec:app:arch']} and Figure \ref{['fig:archdetail']}.
• Figure 3: Speedup via importance prediction, NeRF$^{\dagger}$ vs. our NeRF-ID. Samples deemed to be important by the proposer are kept, different operating points are obtained by varying the importance threshold. 'Relative time' is the time spent on rendering the scene relative to using all samples.
• Figure 4: Qualitative results. Comparison of our NeRF-ID versus NeRF$^{\dagger}$. Overall both methods produce good renderings, but the difference is especially apparent in fine details that NeRF$^{\dagger}$ often misses while NeRF-ID reproduces better, such as thin branches, ropes, markings, edges etc.
• Figure 5: What is learnt? (a) A cross section of the Blender: Lego scene, produced by querying the fine network densely on the plane and plotting the color masked by the occupancy. (b) An image from the test set whose camera plane is roughly parallel to the cross section in (a). (c) and (e) For each row of the cross section image, a ray is shot from left to right and proposals (along with coarse samples) are overlaid in green over the cross section image; the samples are displayed at pixel resolution even though in reality they are real-valued. (c) shows the heuristic proposals of our reimplementation NeRF$^{\dagger}$ of vanilla NeRF Mildenhall20, while (e) shows our NeRF-ID learnt proposals. (f) All proposals from (e) are colored in red such that the intensity is proportional to the estimated importance (Section \ref{['sec:importance']}). The heuristic proposals are over-concentrated in a few areas and sometimes undersample the surface closest to the ray origin (d). NeRF-ID proposals rarely miss the closest surface (e), and are much more diverse which provides a more accurate rendering but also a better sampling that facilitates training of the fine network. Importance prediction does a good job at highlighting the the most promising proposals (f).