Neural Two-Level Monte Carlo Real-Time Rendering

TL;DR

This work tackles real-time global illumination by marrying a two-level Monte Carlo estimator with an online neural radiance cache. The first level uses Neural Incident Radiance Cache (NIRC) to rapidly approximate incident radiance, while a second level estimates the residual bias to achieve an unbiased estimator when desired. A biased variant with a Balanced Termination Heuristic (BTH) and a Neural Visibility Cache (NVC) for environment maps are explored, with substantial variance reductions and speedups demonstrated across multiple scenes. The approach leverages tiny fully-fused MLPs, hash-based encodings, and on-line training to deliver real-time performance with strong convergence even in dynamic scenes, often outperforming Neural Radiance Cache (NRC) baselines. Overall, MLMC paired with NIRC provides superior variance reduction and efficiency, paving the way for robust real-time GI in interactive applications.

Abstract

We introduce an efficient Two-Level Monte Carlo (subset of Multi-Level Monte Carlo, MLMC) estimator for real-time rendering of scenes with global illumination. Using MLMC we split the shading integral into two parts: the radiance cache integral and the residual error integral that compensates for the bias of the first one. For the first part, we developed the Neural Incident Radiance Cache (NIRC) leveraging the power of fully-fused tiny neural networks as a building block, which is trained on the fly. The cache is designed to provide a fast and reasonable approximation of the incident radiance: an evaluation takes 2-25x less compute time than a path tracing sample. This enables us to estimate the radiance cache integral with a high number of samples and by this achieve faster convergence. For the residual error integral, we compute the difference between the NIRC predictions and the unbiased path tracing simulation. Our method makes no assumptions about the geometry, materials, or lighting of a scene and has only few intuitive hyper-parameters. We provide a comprehensive comparative analysis in different experimental scenarios. Since the algorithm is trained in an on-line fashion, it demonstrates significant noise level reduction even for dynamic scenes and can easily be combined with other importance sampling schemes and noise reduction techniques.

Figures (14)

• Figure 1: This figure illustrates the application of the Neural Radiance Cache (NRC, left) NRC and our Neural Incident Radiance Cache (NIRC, right) in the biased path tracing. In NRC, a path is traced from the camera $x_{0}$ to surface point $x_{1}$, where it scatters and then terminates at $x_{2}$ in the cache. In contrast, NIRC stops tracing already at $x_{1}$ and estimates radiance using Monte Carlo, as described in \ref{['eq:L_c']}, by sampling scattering rays based on BRDF and querying the NIRC to predict incident radiance. In practice, the exact termination point is determined by different heuristics for both models (see \ref{['sec:heuristics']}).
• Figure 2: The stages of our neural inference pipeline. Initially, requests in the path tracing pass are gathered (1), utilizing multiple buffers to store per-surface parameters along with the incident direction vectors $\omega_{i}$. Next, the per-surface parameters are encoded (2) and aggregated. Finally, the spherical harmonics coefficients for the requested $\omega_{i}$ are calculated and appended in shared memory as input for the multi-layer perceptron inference (3). Our optimized implementation never stores the neural input and output vectors in global memory.
• Figure 3: Scalability and ablation study with the Bistro Exterior scene rendered on an RTX 3080 at 1080p. We show incident lighting computed using conventional path tracing and our NIRC; the latter has been tested with various optimization strategies. The results demonstrate improved scalability of the NIRC method, particularly when it is optimized by exploiting shared surface encodings (1.74$\Uparrow$ improvement), fused spherical harmonics coefficients computation (2.62$\Uparrow$), and direct output forwarding from the shared memory using atomic variables (3.64$\Uparrow$). For the NRC and the naive NIRC implementations, we show extrapolated performance timings after the 6th frame because of memory limitations (light blue and light red).
• Figure 4: We explore the implications of employing $\mathcal{L}^{2}$ and variance-based $\mathcal{L}_{\mathbb{V}}$ loss functions for the neural cache optimization with our two-level MC estimator and parameters $N_{c} = 4$ and $N_{r} = 1$. We compute the FLIP error between our renders, utilizing Neural Incident Radiance Cache, and reference renders every training epoch. The results demonstrate a detrimental effect on the quality of renders of the biased estimator when the $\mathcal{L}_{\mathbb{V}}$ loss function is applied. This is because the variance-based loss does not necessarily minimize the residual $F_r$. The data also shows divergence during training, likely due to issues when estimating $F_r$ accurately, which is required for evaluating $\mathcal{L}_{\mathbb{V}}$.
• Figure 5: Equal-memory footprint comparison between SHs, vMF mixtures, Neural Control Variates NCV and two our neural caches: NIRC with world space multi-level hash encodings and screen space latent features (SS NIRC). This figure presents shading integrand values in selected surfaces with different materials in a set of scenes, displayed as tone-mapped latitude-longitude images. NCV, SHs and vMFs serve as control variates for analytical integral computations. NIRC focuses only on incident radiance on surfaces for use in MLMC estimators. SHs generally struggle with ringing and fail to capture high-frequency signals for specular surfaces. Although vMFs can theoretically adapt to any frequency, the stepwise EM algorithm can fail to optimize all lobes efficiently with limited noisy radiance samples. Neural Control Variates can evaluate integrals in closed form, but, as evidenced, the expressive power of NIRC demonstrates higher capacity. The average relative variance of the residual error estimator $\mathbb{V}_{rel}(\langle F_{r} \rangle)$ per each scene and method is presented under the images. The plots on the right show estimated $\mathbb{V}_{rel}(\langle F_{r} \rangle)$ using Exponential Moving Average (EMA) per training frame for the both neural models. As illustrated the NIRC requires significantly fewer training frames to achieve even lower $\mathbb{V}_{rel}(\langle F_{r} \rangle)$ than the NCV.