PIR: Photometric Inverse Rendering with Shading Cues Modeling and Surface Reflectance Regularization

TL;DR

This work proposes a new method for neural inverse rendering that jointly optimizes the light source position to account for the self-shadows in images, and computes indirect illumination using a differentiable rendering layer and an importance sampling strategy.

Abstract

This paper addresses the problem of inverse rendering from photometric images. Existing approaches for this problem suffer from the effects of self-shadows, inter-reflections, and lack of constraints on the surface reflectance, leading to inaccurate decomposition of reflectance and illumination due to the ill-posed nature of inverse rendering. In this work, we propose a new method for neural inverse rendering. Our method jointly optimizes the light source position to account for the self-shadows in images, and computes indirect illumination using a differentiable rendering layer and an importance sampling strategy. To enhance surface reflectance decomposition, we introduce a new regularization by distilling DINO features to foster accurate and consistent material decomposition. Extensive experiments on synthetic and real datasets demonstrate that our method outperforms the state-of-the-art methods in reflectance decomposition.

Figures (17)

• Figure 1: Reconstructed 3D assets inserted in a real game scene.
• Figure 1: Quantitative comparison of novel view rendering results with other inverse rendering methods on the synthetic dataset.
• Figure 2: Method overview. Our method optimizes the light source position to account for self-shadows and model inter-reflection. The DINO features are injected into the networks of specular albedo and roughness to regularize the material decomposition.
• Figure 3: Visual results of self-shadows and inter-reflections.
• Figure 4: Qualitative comparison of state-of-the-art methods and our method on the synthetic dataset. The materials of NeILF++zhang2023neilf++ are Base Color, Metallic, Roughness defined by simplified Disney principled BRDF and others are using Mitsuba roughplastic model.