Flash Cache: Reducing Bias in Radiance Cache Based Inverse Rendering

TL;DR

This work uses two techniques to reduce variance for unbiased estimators of the rendering equation: an occlusion-aware importance sampler for incoming illumination and a fast cache architecture that can be used as a control variate for the radiance from a high-quality, but more expensive, volumetric cache.

Abstract

State-of-the-art techniques for 3D reconstruction are largely based on volumetric scene representations, which require sampling multiple points to compute the color arriving along a ray. Using these representations for more general inverse rendering -- reconstructing geometry, materials, and lighting from observed images -- is challenging because recursively path-tracing such volumetric representations is expensive. Recent works alleviate this issue through the use of radiance caches: data structures that store the steady-state, infinite-bounce radiance arriving at any point from any direction. However, these solutions rely on approximations that introduce bias into the renderings and, more importantly, into the gradients used for optimization. We present a method that avoids these approximations while remaining computationally efficient. In particular, we leverage two techniques to reduce variance for unbiased estimators of the rendering equation: (1) an occlusion-aware importance sampler for incoming illumination and (2) a fast cache architecture that can be used as a control variate for the radiance from a high-quality, but more expensive, volumetric cache. We show that by removing these biases our approach improves the generality of radiance cache based inverse rendering, as well as increasing quality in the presence of challenging light transport effects such as specular reflections.

Figures (12)

• Figure 1: Results on two captured scenes from the Stanford ORB dataset StanfordOrb. Our model synthesizes realistic novel views of real-world objects that exhibit complicated specular materials, and recovers accurate material roughnesses (black represents a perfect mirror, white is perfectly rough) and albedo maps.
• Figure 2: Our pipeline for physically-based inverse rendering combines volume rendering with physically-based Monte Carlo rendering to recover geometry, materials, and lighting. We first use an estimator (Equation \ref{['eqn:fast-volume-render']}) to approximate the volume rendering integral (Equation \ref{['eqn:volume-rendering']}) with only a single sample point per ray. Given this sample point, we use a spatially-varying mixture-of-vMFs (Section \ref{['sec:method-low-variance']}) to importance-sample directions based on the distribution of incoming radiance. We then produce a low-variance estimate of the rendering equation (Equation \ref{['eqn:rendering-equation']}) using a NeRF as a radiance cache, and a fast cache approximation of the NeRF (Equation \ref{['eqn:fast-cache']}) that is far less expensive to evaluate.
• Figure 3: An illustration of our two radiance cache architectures. (a) The NeRF-based cache, which is accurate yet expensive to evaluate, uses proposal MLPs to predict sample point distances $\{t_k\}$ through an iterative resampling procedure. These distances are then used to generate sample points at which we query density and outgoing radiance. These quantities are combined to approximate the volume rendering integral via quadrature \ref{['eqn:quadrature']}. (b) Rather than predict sample point distances via proposal sampling, our fast cache directly outputs distances $\{t'_k\}$ as well as weights $\{w'_k\}$ given incoming ray parameters $(\mathbf{x}, \boldsymbol{\omega}_\mathrm{i})$. The distances are used to generate sample points at which we query features from an NGP grid. The resulting features are averaged together, and then fed through a deferred shading MLP which predicts the final color for the incoming ray. Though our illustration depicts four samples in both cases, the NeRF-based cache (a) uses $64$ samples for the first two resampling rounds and $32$ samples for the final one, while our fast cache only uses $8$ samples overall.
• Figure 4: An illustration of our occlusion-aware importance sampler. We use an NGP to predict the parameters of a mixture of vMFs distribution for each point in space (Equation \ref{['eqn:vmfs-ngp']}), which can produce different lobes for different light sources. Because the parameters of these distributions are spatially-varying, if a light source is occluded from the perspective of one surface point, then the model can "turn off" the lobe for that light source.
• Figure 5: (left) We choose two points on the hotdog for which to visualize distributions of incoming radiance. (right) We show (a) incoming radiance from the NeRF cache, which is high quality but slow to evaluate, (b) incoming radiance from the fast cache, which is slightly lower quality but fast to evaluate, and (c) the probability density of the spatially-varying mixture of vMFs, which is able to capture occlusions in the incoming radiance distribution.