PBIR-NIE: Glossy Object Capture under Non-Distant Lighting

TL;DR

This work introduces PBIR-NIE, a physics-based inverse rendering framework that jointly recovers geometry, materials, and non-distant lighting for glossy objects from multi-view images. It couples neural implicit evolution (NIE) with a lightweight, parallax-aware Envmap++ lighting representation to model near-field background illumination and allow topology changes during optimization. A modified antithetic sampling strategy reduces gradient variance for highly glossy BRDFs, enabling stable and fast convergence. The approach delivers state-of-the-art geometry, relighting, and material estimation for glossy objects, validated on synthetic data and real-world Stanford-ORB scenes, and enables realistic view synthesis and lighting-aware reconstructions.

Abstract

Glossy objects present a significant challenge for 3D reconstruction from multi-view input images under natural lighting. In this paper, we introduce PBIR-NIE, an inverse rendering framework designed to holistically capture the geometry, material attributes, and surrounding illumination of such objects. We propose a novel parallax-aware non-distant environment map as a lightweight and efficient lighting representation, accurately modeling the near-field background of the scene, which is commonly encountered in real-world capture setups. This feature allows our framework to accommodate complex parallax effects beyond the capabilities of standard infinite-distance environment maps. Our method optimizes an underlying signed distance field (SDF) through physics-based differentiable rendering, seamlessly connecting surface gradients between a triangle mesh and the SDF via neural implicit evolution (NIE). To address the intricacies of highly glossy BRDFs in differentiable rendering, we integrate the antithetic sampling algorithm to mitigate variance in the Monte Carlo gradient estimator. Consequently, our framework exhibits robust capabilities in handling glossy object reconstruction, showcasing superior quality in geometry, relighting, and material estimation.

Figures (14)

• Figure 1: Overview of our PBIR-NIE pipeline. Our pipeline takes a set of multi-view images capturing a glossy object and an initial shape as input. It then reconstructs the scene's geometry, material properties, and lighting using a physics-based inverse rendering (PBIR) approach. The iterative refinement process includes: 1) Forward Pass: Rendering an image by employing physics-based differentiable rendering. This involves using an explicit mesh extracted with a non-differentiable Marching Cubes algorithm to represent the neural implicit surface for shape, and material networks for surface properties, while leveraging information from input training views. Additionally, Envmap++ is utilized for enhanced lighting representation, replacing the standard infinite-distance environment map to handle non-distant background illumination. 2) Backward Pass: Comparing the rendered image to the ground truth and computing gradients with respect to scene parameters. We use neural implicit evolution (NIE) mehtaLevelSetTheory2022 to facilitate the backpropagation of gradients from the extracted mesh to the neural implicit surface, bypassing the non-differentiable extraction step. 3) Update: Adjusting scene parameters (geometry, material, lighting) via backpropagation to minimize the difference between the rendered and ground truth image.
• Figure 2: Neural Implicit Evolution (NIE). Here we illustrate our PBIR-NIE pipeline for optimizing the underlying geometry using neural implicit evolution (NIE) mehtaLevelSetTheory2022. We represent the geometry with a neural signed distance field (SDF) and employ a surface extraction algorithm, such as non-differentiable marching cubes, to obtain a discretized surface mesh. Next, we compute mesh vertex gradients using physics-based differentiable rendering (PBDR) and update the neural SDF through NIE with the obtained velocity field.
• Figure 3: Antithetic Sampling. When dealing with highly glossy objects (such as a chrome ball in Fig. \ref{['fig:rgb_as']}), traditional BSDF sampling techniques may result in high variance gradients. In Fig. \ref{['fig:as_fd']}, we compute finite differences and display the ground-truth gradient image corresponding to the shape translation of a glossy object with a low roughness value of 0.05. Without antithetic sampling (Fig. \ref{['fig:as_ad']}), the gradient image appears noisy, leading to unstable training. However, by applying antithetic sampling (Fig. \ref{['fig:as_ad_anti']}), we achieve a significantly more reliable Monte Carlo gradient estimation with the same number of samples.
• Figure 4: Our results on Stanford-ORB kuang2024stanford dataset.
• Figure 5: Ablation on Envmap++. We evaluate the quality of glossy object appearance acquisition under non-distant background illumination using our proposed Envmap++ vs. standard environment map lighting.