Extracting Triangular 3D Models, Materials, and Lighting From Images

TL;DR

The paper tackles inverse rendering from multi-view images by producing an explicit triangle mesh with spatially varying materials and an HDR environment map, suitable for direct use in traditional graphics pipelines. It extends differentiable marching tetrahedra (DMTet) to operate under 2D supervision, learning topology, volumetric textures encoded via an MLP, and all-frequency lighting through a differentiable split-sum model. The result is a compact, editable representation that enables scene editing, relighting, and high-quality view interpolation with interactive performance in triangle-based renderers. This approach bridges the gap between neural appearance modeling and production-ready geometry, offering practical workflows and broad applicability in graphics and AR/VR pipelines.

Abstract

We present an efficient method for joint optimization of topology, materials and lighting from multi-view image observations. Unlike recent multi-view reconstruction approaches, which typically produce entangled 3D representations encoded in neural networks, we output triangle meshes with spatially-varying materials and environment lighting that can be deployed in any traditional graphics engine unmodified. We leverage recent work in differentiable rendering, coordinate-based networks to compactly represent volumetric texturing, alongside differentiable marching tetrahedrons to enable gradient-based optimization directly on the surface mesh. Finally, we introduce a differentiable formulation of the split sum approximation of environment lighting to efficiently recover all-frequency lighting. Experiments show our extracted models used in advanced scene editing, material decomposition, and high quality view interpolation, all running at interactive rates in triangle-based renderers (rasterizers and path tracers). Project website: https://nvlabs.github.io/nvdiffrec/ .

Figures (30)

• Figure 1: We reconstruct a triangular mesh with unknown topology, spatially-varying materials, and lighting from a set of multi-view images. We show examples of scene manipulation using off-the-shelf modeling tools, enabled by our reconstructed 3D model.
• Figure 2: Overview of our approach. We learn topology, materials, and environment map lighting jointly from 2D supervision. We leverage differentiable marching tetrahedrons to directly optimize topology of a triangle mesh. While the topology is drastically changing, we learn materials through volumetric texturing, efficiently encoded using an MLP with positional encoding. Finally, we introduce a differentiable version of the split sum approximation for environment lighting. Our output representation is a triangle mesh with spatially varying 2D textures and a high dynamic range environment map, which can be used unmodified in standard game engines. The system is trained end-to-end, supervised by loss in image space, with gradient-based optimization of all stages. Spot model by Keenan Crane.
• Figure 3: Triangle mesh extraction from a set of 256 rendered images w/ masks. Damicornis model from the Smithsonian 3D repository Smithsonian2020, We extracted meshes from NeRF and NeuS using Marching Cubes for a target triangle count of roughly 50k triangles and optimized the example in our pipeline for a similar count. We show renderings of the extracted meshes in a path tracer and report the Chamfer loss. We note that NeuS, which optimizes a surface representation, significantly improves on the volumetric representation used by NeRF for this example. Furthermore, our end-to-end optimization of a triangle mesh improves both the visual quality and the Chamfer loss at a fixed triangle count. When drastically increasing the triangle count in the NeuS mesh extraction (from 53k to 900k triangles), the quality improves significantly, indicating that NeuS has a high quality internal surface representation. Still, our mesh with 53k triangles is on par with the high resolution NeuS output, indicating the benefit of directly optimizing the mesh representation.
• Figure 4: Marching Tetrahedra extracts faces from a tetrahedral grid with grid vertices, $v_i' = v_i + \Delta v_i$ and scalar SDF values, $s_i$. For tets with $\text{sign}(s_i) \neq \text{sign}(s_j)$, faces are extracted, and the face vertices, $v_{ij}$, are determined by by linear interpolation.
• Figure 5: We represent 3D models as a triangular mesh and a set of spatially varying materials following a standard PBR model.