A Convolutional Neural Deferred Shader for Physics Based Rendering

TL;DR

This work tackles photorealistic shading and relighting under real-world illumination by replacing dense MLP shading with a CNN-based neural deferred shader (PBNDS+). It regresses the hemispherical lighting integral using a UNet-style architecture that operates on PBR textures and HDRI illumination, complemented by an energy regularization term to improve dark-condition rendering. The approach outperforms classical models, neural shading baselines, and diffusion-based methods across shading and relighting tasks, while demonstrating robust generalization to unseen HDRIs. Limitations include the absence of volumetric scattering and refraction, with future work aimed at incorporating these effects for truly comprehensive shading.

Abstract

Recent advances in neural rendering have achieved impressive results on photorealistic shading and relighting, by using a multilayer perceptron (MLP) as a regression model to learn the rendering equation from a real-world dataset. Such methods show promise for photorealistically relighting real-world objects, which is difficult to classical rendering, as there is no easy-obtained material ground truth. However, significant challenges still remain the dense connections in MLPs result in a large number of parameters, which requires high computation resources, complicating the training, and reducing performance during rendering. Data driven approaches require large amounts of training data for generalization; unbalanced data might bias the model to ignore the unusual illumination conditions, e.g. dark scenes. This paper introduces pbnds+: a novel physics-based neural deferred shading pipeline utilizing convolution neural networks to decrease the parameters and improve the performance in shading and relighting tasks; Energy regularization is also proposed to restrict the model reflection during dark illumination. Extensive experiments demonstrate that our approach outperforms classical baselines, a state-of-the-art neural shading model, and a diffusion-based method.

Figures (5)

• Figure 1: The overall pipeline of our convolutional neural deferred shading. Given an input image we estimate PBR textures (A: albedo, N: Normal, S: specular, R: roughness, D: depth), IBL lightmap, and field of view, using pre-trained models. Then the estimated data are used to train the convolutional neural deferred shader
• Figure 2: Structure of input tensor $X_{G}$. As 2D convolution only access three dimensional tensors, we combine the batch and light dimension for compatibility, thus each batch instance indicates the contribution of single sampled light ray.
• Figure 3: Quality comparison between different shading models.
• Figure 4: Comparing the result of energy regularization.
• Figure 5: Comparing the result of relighting experiment.