Differentiable Rendering with Reparameterized Volume Sampling

TL;DR

This paper introduces reparameterized volume sampling (RVS), an end-to-end differentiable method for sampling points along rays according to the density-induced distribution in neural radiance fields. By reexpressing the rendering integral with an inverse CDF and employing opacity-based inverses, the authors derive unbiased Monte Carlo radiance estimates and differentiable gradient propagation through the sampling process. They apply RVS to two fronts: (i) a Monte Carlo rendering approach that reduces the number of radiance evaluations per ray, and (ii) a modification of NeRF's hierarchical sampling that eliminates auxiliary proposal losses while improving reconstruction quality. Empirically, RVS improves hierarchy-based NeRF reconstructions across several datasets and enables controllable speed-accuracy tradeoffs in Monte Carlo rendering, with end-to-end training showing clear benefits for hierarchical sampling and practical benefits for fast, inference-time rendering.

Abstract

In view synthesis, a neural radiance field approximates underlying density and radiance fields based on a sparse set of scene pictures. To generate a pixel of a novel view, it marches a ray through the pixel and computes a weighted sum of radiance emitted from a dense set of ray points. This rendering algorithm is fully differentiable and facilitates gradient-based optimization of the fields. However, in practice, only a tiny opaque portion of the ray contributes most of the radiance to the sum. We propose a simple end-to-end differentiable sampling algorithm based on inverse transform sampling. It generates samples according to the probability distribution induced by the density field and picks non-transparent points on the ray. We utilize the algorithm in two ways. First, we propose a novel rendering approach based on Monte Carlo estimates. This approach allows for evaluating and optimizing a neural radiance field with just a few radiance field calls per ray. Second, we use the sampling algorithm to modify the hierarchical scheme proposed in the original NeRF work. We show that our modification improves reconstruction quality of hierarchical models, at the same time simplifying the training procedure by removing the need for auxiliary proposal network losses.

Figures (9)

• Figure 1: Novel views of a ship generated with the proposed Monte Carlo radiance estimates. For each ray we estimate density and then compute radiance at a few ray points generated using the ray density. As the above images indicate, render quality gradually improves with the number of ray samples, without visible artifacts at eight points per ray.
• Figure 2: Illustration of opacity inversion. On the left, we approximate density field $\sigma_{\bm{r}}$ with a piecewise constant and a piecewise linear approximation. On the right, we approximate opacity $F_{\bm{r}}(t)$ and compute $F^{-1}_{\bm{r}}(y_f u)$ for $u \sim U[0, 1]$.
• Figure 3: Comparison between renderings of test-set view on the Lego scene (Blender). Rows correspond to different $(N_p, N_f)$ configurations. NeRF baseline has difficulties in reconstructing fine details for $(16, 32)$ and $(32, 64)$ configurations, and some parts remain blurry even in $(64, 128)$ and $(64, 192)$ configurations, while our method already produces realistic reconstruction for $(32, 64)$ configuration.
• Figure 4: Visualizations across 2D slice of proposal and fine densities learnt on the Lego scene in $(32, 64)$ configuration. Brighter pixels correspond to larger density values.
• Figure 5: Test-time rendering quality as a function of rendering speed. Given a scene representation pre-trained with the standard rendering algorithm, we evaluate rendering algorithms at various configurations.