GSDF: 3DGS Meets SDF for Improved Rendering and Reconstruction

TL;DR

GSDF introduces a dual-branch framework that fuses 3D Gaussian Splatting (GS) for fast, view-dependent rendering with neural Signed Distance Fields (SDF) for precise geometry. Three mutual guidance mechanisms—depth-guided ray sampling from GS to SDF, geometry-aware Gaussian density control from SDF to GS, and mutual depth/normal supervision—coherently align rendering with geometry. Empirical results across diverse datasets show improved rendering quality and more complete, detailed reconstructions, along with faster SDF convergence. The approach preserves each branch’s strengths and offers a scalable path to high-quality rendering and robust geometry for applications in robotics, VR/AR, and embodied environments.

Abstract

Presenting a 3D scene from multiview images remains a core and long-standing challenge in computer vision and computer graphics. Two main requirements lie in rendering and reconstruction. Notably, SOTA rendering quality is usually achieved with neural volumetric rendering techniques, which rely on aggregated point/primitive-wise color and neglect the underlying scene geometry. Learning of neural implicit surfaces is sparked from the success of neural rendering. Current works either constrain the distribution of density fields or the shape of primitives, resulting in degraded rendering quality and flaws on the learned scene surfaces. The efficacy of such methods is limited by the inherent constraints of the chosen neural representation, which struggles to capture fine surface details, especially for larger, more intricate scenes. To address these issues, we introduce GSDF, a novel dual-branch architecture that combines the benefits of a flexible and efficient 3D Gaussian Splatting (3DGS) representation with neural Signed Distance Fields (SDF). The core idea is to leverage and enhance the strengths of each branch while alleviating their limitation through mutual guidance and joint supervision. We show on diverse scenes that our design unlocks the potential for more accurate and detailed surface reconstructions, and at the meantime benefits 3DGS rendering with structures that are more aligned with the underlying geometry.

Figures (8)

• Figure 1: Conceptual Illustration of GSDF. Rendering and reconstruction tasks have traditionally involved trade-offs in neural representation methods. While 3D-GS achieves high-fidelity view-dependent rendering, it often compromises on geometric accuracy. Recent approaches Huang20242DGSguedon2023sugar use explicit regularization to align Gaussian primitives near surfaces, but this can reduce model capacity for high-fidelity visuals. Our GSDF introduces a dual-branch framework with specialized GS- and SDF-branches for rendering and geometry tasks. We propose three mutual guidances (detailed in Sec. \ref{['sec:dual_branch']}) to enhance the quality of both tasks.
• Figure 2: Overview of Dual-branch Guidance. Our dual-branch framework includes a GS-branch for rendering and an SDF-branch for learning neural surfaces. This design preserves the efficiency and fidelity of Gaussian primitive for rendering kerbl20233dscaffoldgs while accurately approximating scene surfaces from an SDF field adapted from NeuS wang2021neus. Specifically: (1) The GS-branch renders depth maps to guide SDF-branch ray sampling, querying absolute SDF values $|s|$ and sampling points within $2k|s|$ (e.g., $k=4$). (2) Predicted SDF values guide GS-branch density control, growing Gaussians near surfaces and pruning deviated ones. (3) Mutual geometry consistency is enforced by comparing depth and normal maps from both branches, ensuring coherent alignment between Gaussians and surfaces.
• Figure 3: Qualitative comparisons of GSDF against popular Gaussian-based baselines kerbl20233dscaffoldgsHuang20242DGS across diverse 3D scene datasets Knapitsch2017barron2022mipDeepBlending2018. As highlighted, GSDF excels in modeling delicate geometries (1st & 2nd rows) and handling texture-less and sparsely observed regions (3rd & 4th rows), which are commonly presented in larger scenes where baseline approaches struggle to address.
• Figure 4: Reconstruction Comparison. We visualize the reconstructed meshes from Instant-NSR instant-nsr-pl (our SDF-branch), SuGaR guedon2023sugar, 2D-GS Huang20242DGS, and Ours.
• Figure 5: Ablation results. Visualizations of reconstructed meshes and rendered images from 1) our full method, 2) ours w/o depth-guided ray sampling, 3) ours w/o geometry-aware density control, and 4) ours w/o geometric supervision. We highlight the degradation of quality using numbered patches.