Learning to Importance Sample in Primary Sample Space

TL;DR

A novel importance sampling technique that uses a neural network to learn how to sample from a desired density represented by a set of samples is proposed that is agnostic of underlying light transport effects, and can be combined with an existing rendering technique by treating it as a black box.

Abstract

Importance sampling is one of the most widely used variance reduction strategies in Monte Carlo rendering. In this paper, we propose a novel importance sampling technique that uses a neural network to learn how to sample from a desired density represented by a set of samples. Our approach considers an existing Monte Carlo rendering algorithm as a black box. During a scene-dependent training phase, we learn to generate samples with a desired density in the primary sample space of the rendering algorithm using maximum likelihood estimation. We leverage a recent neural network architecture that was designed to represent real-valued non-volume preserving ('Real NVP') transformations in high dimensional spaces. We use Real NVP to non-linearly warp primary sample space and obtain desired densities. In addition, Real NVP efficiently computes the determinant of the Jacobian of the warp, which is required to implement the change of integration variables implied by the warp. A main advantage of our approach is that it is agnostic of underlying light transport effects, and can be combined with many existing rendering techniques by treating them as a black box. We show that our approach leads to effective variance reduction in several practical scenarios.

Figures (14)

• Figure 1: Comparison of the conventional approach and ours: (a) Usually, primary sample space (PSS) is sampled uniformly. Each point $y$ in primary sample space (PSS) of dimension $2(k+1)$ corresponds to a geometric path $x$ of length $k$ via a mapping $\Phi_k$. Conventional importance sampling is accounted for by the determinant of the Jacobian of $\Phi_k$. (b) Our approach introduces a non-linear warp $\Psi_k$ in primary sample space to further reduce variance, and we learn this mapping using a neural network.
• Figure 2: We illustrate how we learn the inverse warp $\Psi^{-1}_k(y;\theta)$ to match a target density. (a) We first obtain uniform samples in PSS. (b) We resample them to obtain samples from the target density. (c) The inverse warp $\Psi^{-1}_k(y;\theta)$ is computed to maximize the likelihood of the target samples under the warp.
• Figure 3: Computational structure of our neural importance sampling model based on Real NVP. The core element of this architecture consists of a set of stacked, invertible coupling layers. The functions $s$ and $t$ are implemented using neural networks as shown in Figure \ref{['fig:stnet']}, and stacking several coupling layers makes it possible to represent complex, bijective mappings.
• Figure 4: Element-wise masking of a 4D input vector. Masks are designed as a binary "checkerboard" mode, where elements of the mask are set to alternating 0 and 1.
• Figure 5: The neural network architecture to compute the $s$ and $t$ functions in a bijective coupling layer consists of $N$ residual blocks (RB), containing fully connected layers (FC), batch normalization layers (BN), and ReLU activation units. Additionally, we have fully connected, batch normalization, and activation layers before and after the residual blocks. All fully connected layers contain the same number of neurons.