Real-time Rendering with a Neural Irradiance Volume

TL;DR

Neural Irradiance Volume is proposed, a neural-based technique that allows accurate real-time rendering of diffuse global illumination via a compact pre-computed model, overcoming the limitations of traditional probe-based methods, such as the expensive memory footprint, aliasing artifacts, and scene-specific heuristics.

Abstract

Rendering diffuse global illumination in real-time is often approximated by pre-computing and storing irradiance in a 3D grid of probes. As long as most of the scene remains static, probes approximate irradiance for all surfaces immersed in the irradiance volume, including novel dynamic objects. This approach, however, suffers from aliasing artifacts and high memory consumption. We propose Neural Irradiance Volume (NIV), a neural-based technique that allows accurate real-time rendering of diffuse global illumination via a compact pre-computed model, overcoming the limitations of traditional probe-based methods, such as the expensive memory footprint, aliasing artifacts, and scene-specific heuristics. The key insight is that neural compression creates an adaptive and amortized representation of irradiance, circumventing the cubic scaling of grid-based methods. Our superior memory-scaling improves quality by at least 10x at the same memory budget, and enables a straightforward representation of higher-dimensional irradiance fields, allowing rendering of time-varying or dynamic effects without requiring additional computation at runtime. Unlike other neural rendering techniques, our method works within strict real-time constraints, providing fast inference (around 1 ms per frame on consumer GPUs at full HD resolution), reduced memory usage (1-5 MB for medium-sized scenes), and only requires a G-buffer as input, without expensive ray tracing or denoising.

Figures (21)

• Figure 1: In many scenes, indirect lighting contributes the majority of visible radiance. By caching indirect lighting using an irradiance volume---such as our neural irradiance volume---the significant cost of Monte Carlo integration at runtime can be avoided.
• Figure 2: Diffuse global illumination is inexpensive to render starting from indirect irradiance $E$. However, the indirect irradiance signal itself is expensive to compute, making it the most valuable component to cache. Multiplying the latter by surface albedo $\rho$ results in indirect illumination, and adding direct illumination $D$ and emitted radiance $L_e$ results in diffuse global illumination.
• Figure 3: Varying the hash table size on Sponza (8 levels of encoding) impacts the reconstruction error of NIV. Allowing hash collisions mildly impacts the MSE while significantly reducing the required memory.
• Figure 4: Culling training data inside scene geometry and allocating a portion of the training budget to sampling scene surfaces both improve reconstruction quality. Combining these strategies produces the most robust results.
• Figure 5: Horizontal slice through the banners of Sponza (5.4 MB budget). NIV captures irradiance bleed and shadows better than probe-based methods. NIV has $~\sim10$x higher quality across the scene's volume. Ray tracing the visibility towards probes during evaluation ("+ RT") reduces the overall error but adds a significant performance overhead. The error map highlights the per-pixel absolute luminance error.