DN-Splatter: Depth and Normal Priors for Gaussian Splatting and Meshing

TL;DR

This work tackles the challenge of high-fidelity indoor 3D reconstruction with Gaussian splatting by introducing depth and normal priors. It couples an edge-aware depth loss and monocular normal cues with Gaussian-based priors, enabling stable optimization and direct mesh extraction via Poisson reconstruction. Experiments on MuSHRoom and ScanNet++ show improved novel-view synthesis, depth accuracy, and mesh quality compared to NeRF-, SDF-, and prior Gaussian methods, with ablations highlighting the key role of depth supervision. The approach leverages readily available sensor depth and monocular normals to make indoor 3D reconstruction more accurate and practically deployable for VR/AR pipelines.

Abstract

High-fidelity 3D reconstruction of common indoor scenes is crucial for VR and AR applications. 3D Gaussian splatting, a novel differentiable rendering technique, has achieved state-of-the-art novel view synthesis results with high rendering speeds and relatively low training times. However, its performance on scenes commonly seen in indoor datasets is poor due to the lack of geometric constraints during optimization. In this work, we explore the use of readily accessible geometric cues to enhance Gaussian splatting optimization in challenging, ill-posed, and textureless scenes. We extend 3D Gaussian splatting with depth and normal cues to tackle challenging indoor datasets and showcase techniques for efficient mesh extraction. Specifically, we regularize the optimization procedure with depth information, enforce local smoothness of nearby Gaussians, and use off-the-shelf monocular networks to achieve better alignment with the true scene geometry. We propose an adaptive depth loss based on the gradient of color images, improving depth estimation and novel view synthesis results over various baselines. Our simple yet effective regularization technique enables direct mesh extraction from the Gaussian representation, yielding more physically accurate reconstructions of indoor scenes.

Figures (12)

• Figure 1: Overview: We use depth and normal priors obtained from common handheld devices and general-purpose networks to enhance Gaussian splatting reconstruction quality. By regularizing Gaussian positions, local smoothness, and orientations, we demonstrate improvements in novel-view synthesis and achieve more accurate mesh reconstructions on a variety of challenging indoor room datasets.
• Figure 2: Depth gradient vs. monocular normal supervision strategy. (a) We observe that using pseudo normal maps derived from the gradient of rendered depths R3DG2023 for supervision leads to noisy predicted normals compared to (b) normal supervision by estimates from a pretrained (c) Omnidata model eftekhar2021omnidata.
• Figure 3: Mesh reconstruction results. NeRF variants, even with depth supervision, suffer from artefacts and floaters in reconstruction. The Gaussian based methods Splatfacto, SuGaR, and 2DGS are trained on only photometric losses and thus severly struggle to capture the scene geometry in low texture environments. However, adding depth and normal supervision with DN-Splatter greatly aids reconstruction quality.
• Figure 4: Qualitative comparison of depth and RGB renders against a variety of baselines. DN-Splatter achieves the highest novel view synthesis results compared to NeRF, SDF, and Gaussian based methods.
• Figure 5: Qualitative comparison of depth losses: ScanNet++. We observe that the proposed gradient aware $\mathcal{L_{\hat{D}}}$ regularizer obtains the best qualitative results, mitigating uncertainties at edges from the raw iPhone depth captures. Zoom in to see the details.