GS-2M: Material-aware Gaussian Splatting for High-fidelity Mesh Reconstruction

TL;DR

A material-aware optimization framework for high-fidelity mesh reconstruction from multi-view images based on 3D Gaussian Splatting, referred to as GS-2M, which produces reconstruction results comparable to state-of-the-art methods, delivering accurate triangle meshes even for reflective surfaces.

Abstract

We propose a material-aware optimization framework for high-fidelity mesh reconstruction from multi-view images based on 3D Gaussian Splatting, referred to as GS-2M. Previous works handle these tasks separately and struggle to reconstruct highly reflective surfaces, often relying on priors from external models to enhance the decomposition results. Conversely, our method addresses these two problems by jointly optimizing attributes relevant to the quality of rendered depth and normals, maintaining geometric details while being resilient to reflective surfaces. Although contemporary works effectively solve these tasks together, they often employ sophisticated neural components to learn scene properties, which hinders their performance at scale. To further eliminate these neural components, we propose a novel roughness supervision strategy based on multi-view photometric variation. When combined with a carefully designed loss and optimization process, our unified framework produces reconstruction results comparable to state-of-the-art methods, delivering accurate triangle meshes even for reflective surfaces. We validate the effectiveness of our approach with widely used datasets from previous works and qualitative comparisons with state-of-the-art surface reconstruction methods. Project page: https://ndming.github.io/publications/gs2m/.

Figures (9)

• Figure 1: The reconstructed meshes of reflective objects taken from the Shiny Blender Synthetic dataset Verbin2022refnerf, experimented on recent SoTA surface reconstruction methods: 2DGS Huang2DGS2024, GOF Yu2024GOF, and PGSR Chen2025PGSR. Due to the lack of appearance modeling, these methods often sacrifice geometric details for view-dependent effects caused by highly specular materials, resulting in non-watertight or distorted meshes.
• Figure 2: Comparing depth quality when training with $z$-depth (left) and plane depth (right). We compare normal maps derived from rendered depth maps via a Sobel-like operator for better visualization. Training with $z$-depth results in biased and noisy depth values, while plane depth allows for more accurate and consistent distributions of Gaussians to capture geometric details.
• Figure 3: Filtering invalid correspondences in neighboring views (top) and enhanced multi-view constraints with normal consistency (bottom). Correspondence pixel $\mathbf{p}_2$ of neighbor view $\mathcal{C}_2$ is excluded from multi-view loss calculations because its depth value is less than the $z$-coordinate of the camera-space point $\mathcal{X}$ back-projected from $\mathbf{p}_0$ in the reference view. We also sample values from normal maps rendered at reference and neighboring views to provoke multi-view normal consistency in high-frequency regions.
• Figure 4: Roughness supervision based on multi-view photometric variation. The photometric variation is quantified with the NCC error $L_\mathrm{NCC}$, and a thresholding value $\lambda_\mathrm{ref}$ is used to penalize or reward the corresponding roughness values sampled from the rendered roughness map of the reference view. As $\lambda_\mathrm{ref}$ increases (bottom right), more and more regions become diffuse, i.e., their multi-view photometric variation is not regarded as inconsistency caused by reflective surfaces.
• Figure 5: Simply relying on $L_\mathrm{NCC}$ to identify multi-view photometric variation results in incorrect roughness supervision (middle) at textureless regions. Replacing these regions with gradient-based patches helps $L_\mathrm{NCC}$ produce more faithful results (right).