Progressive Radiance Distillation for Inverse Rendering with Gaussian Splatting

TL;DR

This work tackles the inherent ambiguity in inverse rendering by introducing progressive radiance distillation, a framework that blends a radiance-field representation with a physically-based renderer via a learnable distillation progress map $α$. The method proceeds in four stages, progressively distilling specular then diffuse components while keeping a radiance-field fallback to stabilize gradients and avoid local minima, ultimately enabling high-quality novel views and reliable relighting. It leverages Gaussian splatting, a Cook-Torrance based physical model, and a deferred shading pipeline to produce a hybrid renderer whose output is a weighted blend of $I_{phy}$ and $I_{raw}$. Experimental results on multiple synthetic and real datasets show state-of-the-art or competitive performance in NVS and relighting, with robust decomposition of normals, albedo, and lighting, and ablations validate the effectiveness of the distillation schedule and loss terms. The approach also demonstrates generalization to mesh-based inverse rendering, indicating broad applicability for efficient, accurate scene relighting and view synthesis.

Abstract

We propose progressive radiance distillation, an inverse rendering method that combines physically-based rendering with Gaussian-based radiance field rendering using a distillation progress map. Taking multi-view images as input, our method starts from a pre-trained radiance field guidance, and distills physically-based light and material parameters from the radiance field using an image-fitting process. The distillation progress map is initialized to a small value, which favors radiance field rendering. During early iterations when fitted light and material parameters are far from convergence, the radiance field fallback ensures the sanity of image loss gradients and avoids local minima that attracts under-fit states. As fitted parameters converge, the physical model gradually takes over and the distillation progress increases correspondingly. In presence of light paths unmodeled by the physical model, the distillation progress never finishes on affected pixels and the learned radiance field stays in the final rendering. With this designed tolerance for physical model limitations, we prevent unmodeled color components from leaking into light and material parameters, alleviating relighting artifacts. Meanwhile, the remaining radiance field compensates for the limitations of the physical model, guaranteeing high-quality novel views synthesis. Experimental results demonstrate that our method significantly outperforms state-of-the-art techniques quality-wise in both novel view synthesis and relighting. The idea of progressive radiance distillation is not limited to Gaussian splatting. We show that it also has positive effects for prominently specular scenes when adapted to a mesh-based inverse rendering method.

Figures (18)

• Figure 1: The rendering model and training stages of our method. We adopt a deferred shading pipeline. First, geometry parameters (position: $\mathbf{x}$, covariance: $\mathbf{\Sigma}$ and opacity: $o$) and shading parameters (albedo: $\boldsymbol{c}$, metallic: $m$, roughness: $r$, SH colors $y_l^m$ and distillation progress: $\alpha$) are splatted into several screen space maps, including material maps (albedo, metallic and roughness), geometry maps (depth and normal), the raw radiance map $\mathrm{I_{raw}}$ and the distillation progress map $\alpha$. Then, we send the maps into a physically-based shader to produce diffuse and specular color maps: $\mathrm{I_{diff}}$ and $\mathrm{I_{spec}}$. The physical term $\mathrm{I_{phy}}$ combines $\mathrm{I_{diff}}$ and $\mathrm{I_{spec}}$, and then gets blended with $\mathrm{I_{raw}}$ using $\alpha$ as weight to produce the final result. The various shading parameters are distilled from $\mathrm{I_{raw}}$ using a four-stage training process. To mitigate ambiguity, we subset and specialize the rendering model for each individual stage.
• Figure 2: Visualization of the distillation progress map and different rendering components generated by each optimization stage.
• Figure 3: Qualitative comparisons of novel view synthesis results. From top to bottom: cactus (from Stanford ORB stanfordorb), luyu and potion (from Glossy Synthetic nero), car and toaster (from Shiny Blender ref_nerf).
• Figure 4: Qualitative comparisons of normal estimated by different methods. From top to bottom: potion (from Glossy Synthetic nero), car (from Shiny Blender ref_nerf) and teapot (from Stanford ORB stanfordorb).
• Figure 5: Qualitative comparisons of environment maps estimated by different methods. From top to bottom: helmet, teapot and toaster (from Shiny Blender ref_nerf), gnome (from Stanford ORB stanfordorb).