GaussianUDF: Inferring Unsigned Distance Functions through 3D Gaussian Splatting

TL;DR

The paper addresses reconstructing open surfaces from multi-view images using unsigned distance functions (UDFs) but faces challenges bridging continuous UDFs with discrete 3D Gaussian splatting. It introduces a novel approach that overfits thin 2D Gaussian planes on surfaces and uses self-supervision together with gradient-based inference to estimate unsigned distances both near and far from the surface, optimizing the UDF $f$ via differentiable 3D Gaussian splatting. The method projects Gaussian centers onto the zero level set of $f$, applies depth and normal regularization, and employs a multi-term loss $L = (1-\lambda_1)L_{rgb} + \lambda_1 L_{ssim} + \lambda_2 L_{far} + \lambda_3 L_{near} + \lambda_4 L_{proj} + \lambda_5 L_{depth} + \lambda_6 L_{norm}$ to guide learning. Experiments on DF3D, DTU, and real scans show state-of-the-art accuracy, completeness, and sharp open surfaces with efficient training thanks to 3D Gaussian splatting.

Abstract

Reconstructing open surfaces from multi-view images is vital in digitalizing complex objects in daily life. A widely used strategy is to learn unsigned distance functions (UDFs) by checking if their appearance conforms to the image observations through neural rendering. However, it is still hard to learn continuous and implicit UDF representations through 3D Gaussians splatting (3DGS) due to the discrete and explicit scene representation, i.e., 3D Gaussians. To resolve this issue, we propose a novel approach to bridge the gap between 3D Gaussians and UDFs. Our key idea is to overfit thin and flat 2D Gaussian planes on surfaces, and then, leverage the self-supervision and gradient-based inference to supervise unsigned distances in both near and far area to surfaces. To this end, we introduce novel constraints and strategies to constrain the learning of 2D Gaussians to pursue more stable optimization and more reliable self-supervision, addressing the challenges brought by complicated gradient field on or near the zero level set of UDFs. We report numerical and visual comparisons with the state-of-the-art on widely used benchmarks and real data to show our advantages in terms of accuracy, efficiency, completeness, and sharpness of reconstructed open surfaces with boundaries.

Figures (11)

• Figure 1: The comparisons with 2DGS huang20242dgs, 2S-UDF deng20242sudf, and VRPrior zhang2024vrprior. Our method recovers the most accurate open surfaces without artifacts.
• Figure 2: Overview of our method. (a) The UDF is optimized with the rendering process. To ensure that Gaussians can provide more accurate clues of the surfaces, (b) the Gaussians are projected to the zero level set of the UDF. (c) Projecting random queries to the Gaussian centers helps the UDF learn coarse distance fields which is far from surfaces. Moreover, (d) unsigned distances recovered near the Gaussian plane compensates for the sparsity of Gaussian centers. We adopt depth (e) and normal (f) regularization terms to make Gaussians align with surfaces well.
• Figure 3: Self-supervision loss. For a Gaussian in (a), (b) we first sample root point $r^h_i$ on the plane. (c) Then we randomly move the root point to position $e^b_{i,h}$ along or against the normal with a randomly sampled offset $t_b$. (d) We use $\{e^b_{i,h}, t_b\}$ as a training sample pair to train the UDF network. (e) The reconstructed meshes show that the 2D Gaussian planes provide more surface information for the UDF, which helps to fill the holes and capture more details.
• Figure 4: We project the Gaussian centers to the zero level set with a constraint, which makes the point cloud have less noises and the UDF have more accurate surface.
• Figure 5: Qualitative comparison with 2DGS huang20242dgs, GOF Yu2024GOF, NeuralUDF long2023neuraludf, 2S-UDF deng20242sudf, and VRPrior zhang2024vrprior in DF3D zhu2020df3d dataset. Note that VRPrior needs additional depth images to learn priors. The dark color on meshes represents the back faces of open surfaces, and the error map is shown next to the mesh. Our method obtains more accurate surfaces and captures more details such as the folds in the clothing.