Neural-PBIR Reconstruction of Shape, Material, and Illumination

TL;DR

Neural-PBIR tackles the long-standing problem of jointly reconstructing an object's shape, spatially varying material, and surrounding illumination from 2D images. It introduces a three-stage pipeline: (1) neural surface reconstruction with a hybrid SDF grid and radiance field for accurate geometry; (2) neural distillation that converts the radiance field into a physics-based material and illumination representation; and (3) physics-based inverse rendering (PBIR) to refine geometry, SVBRDF, and environment lighting using differentiable rendering that accounts for global illumination. The approach delivers state-of-the-art geometry accuracy, material fidelity, and lighting realism on both synthetic and real data, while significantly reducing computation time relative to prior neural-volume methods. This combination enables realistic relighting, novel-view synthesis, and robust scene reconstructions, with broad implications for immersive graphics and vision applications.

Abstract

Reconstructing the shape and spatially varying surface appearances of a physical-world object as well as its surrounding illumination based on 2D images (e.g., photographs) of the object has been a long-standing problem in computer vision and graphics. In this paper, we introduce an accurate and highly efficient object reconstruction pipeline combining neural based object reconstruction and physics-based inverse rendering (PBIR). Our pipeline firstly leverages a neural SDF based shape reconstruction to produce high-quality but potentially imperfect object shape. Then, we introduce a neural material and lighting distillation stage to achieve high-quality predictions for material and illumination. In the last stage, initialized by the neural predictions, we perform PBIR to refine the initial results and obtain the final high-quality reconstruction of object shape, material, and illumination. Experimental results demonstrate our pipeline significantly outperforms existing methods quality-wise and performance-wise.

Figures (20)

• Figure 1: Neural-PBIR recovers high-fidelity material (b1,2), shape and lighting (b3), enabling realistic re-rendering (c1-3).
• Figure 2: Our pipeline for joint shape, material, and lighting estimation.
• Figure 3: Qualitative comparisons on the MII data.
• Figure 4: Novel-view interpolation on our real dataset. Our technique produces high-fidelity reconstructions with minimal artifacts. We report the average PSNR$\uparrow$ and SSIM$\uparrow$ below each image.
• Figure 5: Rerendering of reconstruction results under captured (GT) illumination. We rescale all renderings to match the overall brightness of the GT image.