Fast and Accurate Neural Rendering Using Semi-Gradients

TL;DR

A new objective function is introduced that maintains the same global optimum as before but allows for unbiased and low-variance gradient estimates, enabling faster and more accurate training of neural networks.

Abstract

We propose a simple yet effective neural network-based framework for global illumination rendering. Recently, rendering techniques that learn neural radiance caches by minimizing the difference (i.e., residual) between the left and right sides of the rendering equation have been suggested. Due to their ease of implementation and the advantage of excluding path integral calculations, these techniques have been applied to various fields, such as free-viewpoint rendering, differentiable rendering, and real-time rendering. However, issues of slow training and occasionally darkened renders have been noted. We identify the cause of these issues as the bias and high variance present in the gradient estimates of the existing residual-based objective function. To address this, we introduce a new objective function that maintains the same global optimum as before but allows for unbiased and low-variance gradient estimates, enabling faster and more accurate training of neural networks. In conclusion, this method is simply implemented by ignoring the partial derivatives of the right-hand side, and theoretical and experimental analyses demonstrate the effectiveness of the proposed loss.

Figures (9)

• Figure 1: Example to motivate our partial derivative-based optimization for faster convergence. From the start of training (100 iterations, 14 seconds) of the baseline (Sec. \ref{['preliminaries']}), the RHS quickly converges to the reference, while the LHS lags behind with issues such as faint colors and brightness overshoot on lampshades. As such, we ignore the partial derivative with respect to the RHS, focusing on learning the LHS. For visualization, the LHS and the RHS are shown at 4 and 1024 samples-per-pixel (spp), respectively.
• Figure 2: Equal-iteration comparisons for our method and the baseline approach. The figure displays the convergence of image error, averaged across seven scenes in Fig. \ref{['fig:superior_reconstruction']}. Our method shows an average error that is 8.8 times lower.
• Figure 3: When rendering free-viewpoint video along a camera path in each scene after training, the error of our method is lower than the baseline in all frames. Refer to the supplementary materials for the videos.
• Figure 4: Visual comparison between the baseline (biased gradient estimator), dual-buffer (unbiased gradient estimator) pidhorskyi2022depthdeng2022reconstructing, and our methods. When using the baseline method, the resulting reconstruction is noticeably darker. The dual-buffer approach resolves the bias on gradients yet fails to reconstruct glossy reflections shapely in the same training iterations. Increasing the incident sample count eightfold (to 256), taking 16-fold time, is necessary to achieve successful restoration. Our method succeeds in reconstruction with even less time than the baseline, without the need for such excessive sampling. VR stands for variance reduction.
• Figure 5: Linear interpolation between the gradients of the dual-buffer and our method using weight $w$. The key difference between the two methods is whether the RHS partial derivative is included in the gradient. As the weight is reduced to 0, the influence of the RHS partial derivative diminishes, converging to the semi-gradient method, and the model's performance improves. The figure displays the convergence of image error, averaged across seven scenes in Fig. \ref{['fig:superior_reconstruction']}. VR stands for variance reduction.