Neural Signed Distance Function Inference through Splatting 3D Gaussians Pulled on Zero-Level Set

TL;DR

This work proposes a method that seamlessly merge 3DGS with the learning of neural SDFs, and jointly optimize 3D Gaussians and the neural SDF with both RGB and geometry constraints, which recovers more accurate, smooth, and complete surfaces with more geometry details.

Abstract

It is vital to infer a signed distance function (SDF) in multi-view based surface reconstruction. 3D Gaussian splatting (3DGS) provides a novel perspective for volume rendering, and shows advantages in rendering efficiency and quality. Although 3DGS provides a promising neural rendering option, it is still hard to infer SDFs for surface reconstruction with 3DGS due to the discreteness, the sparseness, and the off-surface drift of 3D Gaussians. To resolve these issues, we propose a method that seamlessly merge 3DGS with the learning of neural SDFs. Our key idea is to more effectively constrain the SDF inference with the multi-view consistency. To this end, we dynamically align 3D Gaussians on the zero-level set of the neural SDF using neural pulling, and then render the aligned 3D Gaussians through the differentiable rasterization. Meanwhile, we update the neural SDF by pulling neighboring space to the pulled 3D Gaussians, which progressively refine the signed distance field near the surface. With both differentiable pulling and splatting, we jointly optimize 3D Gaussians and the neural SDF with both RGB and geometry constraints, which recovers more accurate, smooth, and complete surfaces with more geometry details. Our numerical and visual comparisons show our superiority over the state-of-the-art results on the widely used benchmarks.

Figures (15)

• Figure 1: Overview of our method. We (a) pull 3D Gaussians onto the zero-level set for splatting, while (b) pulling the neighboring space onto the Gaussian disks for SDF inference. To better facilitate this procedure, we introduce three constraints: (c) push the Gaussians to become disks; (d) encourage the disk to be a tangent plane on the zero-level set; (e) constrain the query points to be pulled along the shortest path.
• Figure 2: Comparison of pulling Gaussians to centers and to disks. The former tends to overfit sparse Gaussian centers, resulting in incomplete meshes. We address this issue by pulling queries onto disk planes.
• Figure 3: Visual comparisons on DTU dataset.
• Figure 4: Visual comparisons on Tanks and Temples dataset.
• Figure 5: Visual comparisons on Mip-NeRF 360 dataset.