MatDecompSDF: High-Fidelity 3D Shape and PBR Material Decomposition from Multi-View Images

TL;DR

MatDecompSDF addresses the challenging inverse rendering problem by jointly learning a neural SDF geometry, spatially varying PBR materials, and an environment lighting model, all within a differentiable rendering framework. It introduces physically grounded priors (eikonal, material smoothness, metallic sparsity) to stabilize decomposition and enable editable, relightable assets. The approach yields state-of-the-art geometric and material fidelity on synthetic data and strong relighting performance on real DTU data, while delivering meshes and texture maps ready for standard graphics pipelines. This work advances practical 3D content creation by providing accurate, editable reconstructions that support realistic relighting and material editing from multi-view imagery.

Abstract

We present MatDecompSDF, a novel framework for recovering high-fidelity 3D shapes and decomposing their physically-based material properties from multi-view images. The core challenge of inverse rendering lies in the ill-posed disentanglement of geometry, materials, and illumination from 2D observations. Our method addresses this by jointly optimizing three neural components: a neural Signed Distance Function (SDF) to represent complex geometry, a spatially-varying neural field for predicting PBR material parameters (albedo, roughness, metallic), and an MLP-based model for capturing unknown environmental lighting. The key to our approach is a physically-based differentiable rendering layer that connects these 3D properties to the input images, allowing for end-to-end optimization. We introduce a set of carefully designed physical priors and geometric regularizations, including a material smoothness loss and an Eikonal loss, to effectively constrain the problem and achieve robust decomposition. Extensive experiments on both synthetic and real-world datasets (e.g., DTU) demonstrate that MatDecompSDF surpasses state-of-the-art methods in geometric accuracy, material fidelity, and novel view synthesis. Crucially, our method produces editable and relightable assets that can be seamlessly integrated into standard graphics pipelines, validating its practical utility for digital content creation.

Figures (4)

• Figure 1: An overview of our proposed MatDecompSDF framework. Given multi-view images, we jointly optimize a geometry network ($f_{geo}$), a material network ($f_{mat}$), and a lighting network ($f_{light}$). A key component is our differentiable PBR volume renderer, which computes pixel colors based on the predicted geometry, materials, and lighting. The photometric loss between the rendered and ground-truth images, along with several regularization terms, drives the optimization to disentangle these three components.
• Figure 2: Qualitative comparison of PBR material decomposition on the synthetic dataset. From top to bottom: Ground Truth, our MatDecompSDF, NVDIFFREC munkberg2022nvdiffrec, and Mat-NeuS zhang2022matneus. From left to right: Albedo, Roughness, and Metallic maps. Our method produces significantly cleaner albedo maps, free from the lighting and shadow artifacts present in the baselines, and more accurately recovers roughness and metallic properties.
• Figure 3: Qualitative results on the DTU dataset and demonstration of relighting and editing capabilities. (a) Geometric reconstruction comparison, where our method captures finer details than the baseline. (b) Our reconstructed asset rendered under two novel lighting environments (a warm point light and a cool HDR map), exhibiting physically plausible shading. (c) Material editing demonstration: we modify the albedo map (center) and re-render the object (right), showing the correct appearance update.
• Figure 4: Limitations and failure cases of our method. (Top) For highly transparent objects like glass, our framework fails to model light transport correctly, resulting in a distorted, opaque reconstruction. (Bottom) For materials with strong anisotropic reflectance like brushed metal, our PBR model cannot capture the direction-dependent highlights, leading to a blurry and inaccurate appearance.