Inverse Rendering using Multi-Bounce Path Tracing and Reservoir Sampling

TL;DR

This paper tackles the ill-posed problem of inverse rendering from multi-view images by proposing MIRReS, a two-stage mesh-based framework that explicitly optimizes geometry, materials, and illumination. Stage 1 extracts a coarse mesh from a neural radiance field and NeuS2, while Stage 2 refines the mesh with trainable vertex offsets and jointly optimizes a Disney BRDF-based material model and a learnable environment map under a physically-based rendering pipeline that includes multi-bounce path tracing. Direct lighting is efficiently estimated with reservoir sampling to reduce noise, and indirect illumination is computed via multi-bounce path tracing to capture self-shadowing and inter-reflections, enabling more accurate intrinsic decomposition and relighting. The method achieves state-of-the-art results on both geometry reconstruction and intrinsic decomposition on challenging datasets, and the explicit mesh representation enables downstream tasks like scene editing and relighting with standard graphics tools. The work provides a practical, physically-grounded alternative to implicit neural representations for professional graphics pipelines, with code available to foster adoption.

Abstract

We present MIRReS, a novel two-stage inverse rendering framework that jointly reconstructs and optimizes the explicit geometry, material, and lighting from multi-view images. Unlike previous methods that rely on implicit irradiance fields or simplified path tracing algorithms, our method extracts an explicit geometry (triangular mesh) in stage one, and introduces a more realistic physically-based inverse rendering model that utilizes multi-bounce path tracing and Monte Carlo integration. By leveraging multi-bounce path tracing, our method effectively estimates indirect illumination, including self-shadowing and internal reflections, which improves the intrinsic decomposition of shape, material, and lighting. Moreover, we incorporate reservoir sampling into our framework to address the noise in Monte Carlo integration, enhancing convergence and facilitating gradient-based optimization with low sample counts. Through qualitative and quantitative evaluation of several scenarios, especially in challenging scenarios with complex shadows, we demonstrate that our method achieves state-of-the-art performance on decomposition results. Additionally, our optimized explicit geometry enables applications such as scene editing, relighting, and material editing with modern graphics engines or CAD software. The source code is available at https://brabbitdousha.github.io/MIRReS/

Figures (14)

• Figure 1: Capabilities of MIRRes. Given multi-view images of a 3D scene, our method jointly optimizes geometry, materials and lighting to achieve high-quality reconstructions. This facilitates applications including novel view synthesis, relighting, and scene editing.
• Figure 2: Overview of our inverse rendering pipeline. Our two-stage process starts with the extraction of a coarse mesh from a radiance field, followed by joint optimization of geometry, material, and lighting using physically-based rendering techniques. Key components such as multi-bounce path tracing, Monte Carlo integration, and reservoir sampling are highlighted to showcase their roles in enhancing the accuracy and efficiency of the reconstruction process.
• Figure 3: Comparison on rendering noise with or without reservoir sampling with sample count 1.
• Figure 4: Our rendering results of direct (b), indirect (c), and full (a) lighting in the Lego scene. Note that the sharp light visibility in (d) demonstrates the accuracy of our path-tracing rendering model and our reconstruction geometry.
• Figure 5: Qualitative comparison of the reconstructed mesh on the TensoIR dataset. Zoom in for details.