Differentiable Inverse Rendering with Interpretable Basis BRDFs

TL;DR

This work targets the ill-posed problem of recovering geometry and spatially varying BRDFs from images. It introduces a differentiable inverse rendering pipeline that represents geometry with 2D Gaussians and reflectance with a set of basis BRDFs blended per Gaussian, while dynamically adjusting the basis count and enforcing sparsity to yield interpretable, spatially separated BRDFs. A specular-weighted rendering loss further emphasizes challenging highlights, and basis BRDF merging/removal plus scheduling ensure the basis set adapts to scene complexity. The approach yields accurate geometry, scalable interpretable BRDFs, and supports novel-view relighting and intuitive scene editing, with faster training and superior normal reconstruction compared to state-of-the-art methods. Overall, this method provides a practical, interpretable, and adaptable framework for differentiable inverse rendering of complex scenes.

Abstract

Inverse rendering seeks to reconstruct both geometry and spatially varying BRDFs (SVBRDFs) from captured images. To address the inherent ill-posedness of inverse rendering, basis BRDF representations are commonly used, modeling SVBRDFs as spatially varying blends of a set of basis BRDFs. However, existing methods often yield basis BRDFs that lack intuitive separation and have limited scalability to scenes of varying complexity. In this paper, we introduce a differentiable inverse rendering method that produces interpretable basis BRDFs. Our approach models a scene using 2D Gaussians, where the reflectance of each Gaussian is defined by a weighted blend of basis BRDFs. We efficiently render an image from the 2D Gaussians and basis BRDFs using differentiable rasterization and impose a rendering loss with the input images. During this analysis-by-synthesis optimization process of differentiable inverse rendering, we dynamically adjust the number of basis BRDFs to fit the target scene while encouraging sparsity in the basis weights. This ensures that the reflectance of each Gaussian is represented by only a few basis BRDFs. This approach enables the reconstruction of accurate geometry and interpretable basis BRDFs that are spatially separated. Consequently, the resulting scene representation, comprising basis BRDFs and 2D Gaussians, supports physically-based novel-view relighting and intuitive scene editing.

Figures (11)

• Figure 1: We propose a differentiable inverse rendering method with (a) multi-view flash photography inputs. Our analysis-by-synthesis method achieves not only (b) novel-view relighting and accurate geometry reconstruction, but also (c) interpretable basis BRDFs and their spatially-separated weights. This allows for (d) intuitive scene editing.
• Figure 2: The process of our analysis-by-synthesis iterations. Given a set of multi-view photometric images, we initialize point cloud and extract base color for basis BRDFs. We jointly optimize 2D Gaussians and basis BRDFs by comparing the differentiably-rendered images and the input images. Our method enables obtaining interpretable basis BRDFs with spatially-separated basis-BRDF weights and the number of basis BRDFs adapts to the scene.
• Figure 3: Basis BRDFs with varying scene complexity. We adjust the number of basis BRDFs during analysis-by-synthesis iterations to adapt to the scene complexity. We initialize the same number of basis BRDFs for both scenes in this example.
• Figure 4: Basis BRDF control. During the analysis-by-synthesis optimization, we compute the values of each basis BRDF for sampled half-way angles $\theta_{h}$ from which radiometric difference is obtained. We compute the geometric difference between point clouds of basis BRDFs. If two basis BRDFs are radiometrically and geometrically similar, we merge them. If the rendered weight map $W_i$ has few valid pixels, we remove the basis BRDF.
• Figure 5: Normal reconstruction. Our method successfully recovers the detailed surface normal, outperforming other state-of-the-art inverse rendering methods: IRON iron, DPIR chung2024differentiable, GS3bi2024rgs.