3DGSR: Implicit Surface Reconstruction with 3D Gaussian Splatting

TL;DR

This work tackles accurate 3D surface reconstruction from multi-view data by integrating an implicit signed distance field (SDF) within 3D Gaussian Splatting (3DGS). It introduces two coupling strategies—tight coupling via a differentiable SDF-to-opacity transformation and loose coupling via surface-normal alignment—along with volumetric rendering-based regularization to provide dense supervision for the SDF. The method yields high-fidelity geometry and rendering quality, outperforming neural implicit and Gaussian-based baselines while significantly reducing training time (about 12-30k iterations) and maintaining real-time-ish rendering speeds. By jointly optimizing the SDF and Gaussians, 3DGSR eliminates post-processing steps like Poisson reconstruction and depth fusion, enabling end-to-end high-quality surface reconstruction with efficient inference. The approach also demonstrates robustness to pose noise and can leverage sparse depth cues to further enhance reconstruction in challenging real-world scenarios.

Abstract

In this paper, we present an implicit surface reconstruction method with 3D Gaussian Splatting (3DGS), namely 3DGSR, that allows for accurate 3D reconstruction with intricate details while inheriting the high efficiency and rendering quality of 3DGS. The key insight is incorporating an implicit signed distance field (SDF) within 3D Gaussians to enable them to be aligned and jointly optimized. First, we introduce a differentiable SDF-to-opacity transformation function that converts SDF values into corresponding Gaussians' opacities. This function connects the SDF and 3D Gaussians, allowing for unified optimization and enforcing surface constraints on the 3D Gaussians. During learning, optimizing the 3D Gaussians provides supervisory signals for SDF learning, enabling the reconstruction of intricate details. However, this only provides sparse supervisory signals to the SDF at locations occupied by Gaussians, which is insufficient for learning a continuous SDF. Then, to address this limitation, we incorporate volumetric rendering and align the rendered geometric attributes (depth, normal) with those derived from 3D Gaussians. This consistency regularization introduces supervisory signals to locations not covered by discrete 3D Gaussians, effectively eliminating redundant surfaces outside the Gaussian sampling range. Our extensive experimental results demonstrate that our 3DGSR method enables high-quality 3D surface reconstruction while preserving the efficiency and rendering quality of 3DGS. Besides, our method competes favorably with leading surface reconstruction techniques while offering a more efficient learning process and much better rendering qualities. The code will be available at https://github.com/CVMI-Lab/3DGSR.

Figures (9)

• Figure 1: Oversmooth reconstruction results from 2DGS Huang2DGS2024 lead to poorer performance, reflected by a higher chamfer-$\mathcal{L}_1$ error (1.40) compared to 3DGSR (0.72).
• Figure 2: Imperfect reconstruction from the non-uniform distributions of Gaussians. Directly defining the surface from the Gaussian points may lead the noisy reconstruction or artifacts.
• Figure 3: Pipeline of our proposed approach for implicit surface reconstruction. We model the surface with an implicit SDF field, with which the SDF value of each 3D Gaussian can be predicted. We propose two different coupling strategies to make the distribution of Gaussians align with the implicit SDF field. The geometry attributes of 3D Gaussians serve as a regularization for the SDF field, while the rendered image is supervised by the captured image.
• Figure 4: Qualitative comparisons of different coupling methods and the influence of geometric constraint.
• Figure 5: Qualitative comparisons of reconstructed results between Relightable 3D Gaussian gao2023relightable, 2DGS Huang2DGS2024 and our method on the DTU dataset. Our method yields the highest-quality surface reconstruction preserving fine details and intricate structures.