Baking Relightable NeRF for Real-time Direct/Indirect Illumination Rendering

TL;DR

A novel method that executes a CNN renderer to compute primary surface points and rendering parameters, required for direct illumination, is proposed and a lightweight hash grid-based renderer is presented, for indirect illumination, which is recursively executed to perform the secondary ray tracing process.

Abstract

Relighting, which synthesizes a novel view under a given lighting condition (unseen in training time), is a must feature for immersive photo-realistic experience. However, real-time relighting is challenging due to high computation cost of the rendering equation which requires shape and material decomposition and visibility test to model shadow. Additionally, for indirect illumination, additional computation of rendering equation on each secondary surface point (where reflection occurs) is required rendering real-time relighting challenging. We propose a novel method that executes a CNN renderer to compute primary surface points and rendering parameters, required for direct illumination. We also present a lightweight hash grid-based renderer, for indirect illumination, which is recursively executed to perform the secondary ray tracing process. Both renderers are trained in a distillation from a pre-trained teacher model and provide real-time physically-based rendering under unseen lighting condition at a negligible loss of rendering quality.

Figures (9)

• Figure 1: Overview of the rendering process of our proposed baked model.
• Figure 2: Architecture of our proposed CNN-based renderer. Instance normalization in with affine parameters and GELU gelu are used for all normalization and activation layers, respectively. (a): Our direct illumination renderer consists of an initial 1x1 conv-norm-act layer, a deep residual MLP, and three super-resolution blocks. Each output from an SR block is bilinearly upsampled and added to the output from the next SR block. (b): Inside an SR block, an input feature map is upsampled bilinearly and processed with two consecutive 3x3 conv-norm-act layers. Then, the output feature map is fed to the output mapping layers (1x1 convolutions) to obtain rendering parameters.
• Figure 3: Architecture of our proposed hash grid-based renderer for visibility modeling and indirect illumination.
• Figure 4: Quantitative results on all 6 synthetic scenes. Environment lights and viewpoints are randomly selected for each scene.
• Figure 5: Visualization of albedo maps with floater artifacts extracted from TensoIR-MIS teachers.