Geometry Field Splatting with Gaussian Surfels

TL;DR

This work introduces Geometry Field Splatting with Gaussian Surfels to reconstruct opaque surfaces from calibrated RGB images by modeling a stochastic geometry field and converting it to a density for differentiable rendering. It derives an almost exact rendering pipeline that eliminates key approximations, and it remedies loss-landscape discontinuities by enforcing continuous color behavior across overlapping kernels, while exploring latent color representations to better capture specularities. The approach yields substantial improvements in geometric fidelity on standard datasets (DTU, BlendedMVS, Mip-NeRF 360) over neural and prior splatting methods, with efficient rendering enabled by Gaussian surfels and a refined splatting formulation. Limitations include handling only opaque objects and potential challenges for transparent or highly fuzzy surfaces, with future work pointing to anti-aliasing, appearance embeddings, and improved mesh extraction.

Abstract

Geometric reconstruction of opaque surfaces from images is a longstanding challenge in computer vision, with renewed interest from volumetric view synthesis algorithms using radiance fields. We leverage the geometry field proposed in recent work for stochastic opaque surfaces, which can then be converted to volume densities. We adapt Gaussian kernels or surfels to splat the geometry field rather than the volume, enabling precise reconstruction of opaque solids. Our first contribution is to derive an efficient and almost exact differentiable rendering algorithm for geometry fields parameterized by Gaussian surfels, while removing current approximations involving Taylor series and no self-attenuation. Next, we address the discontinuous loss landscape when surfels cluster near geometry, showing how to guarantee that the rendered color is a continuous function of the colors of the kernels, irrespective of ordering. Finally, we use latent representations with spherical harmonics encoded reflection vectors rather than spherical harmonics encoded colors to better address specular surfaces. We demonstrate significant improvement in the quality of reconstructed 3D surfaces on widely-used datasets.

Figures (16)

• Figure 1: We introduce a geometry reconstruction method from a set of calibrated RGB images by splatting a geometry field with Gaussian surfels. We enable almost exact and efficient differentiable rendering. Compared to 2DGS, we reach better geometry quality and achieve overall smooth and detailed geometry without having cracks or holes. The reader may wish to zoom into the electronic version in the figures. (Note: object colors used are for visualization, not original object color).
• Figure 2: Illustration of a ray intersecting with non-overlapping kernels $\mathcal{K}_1, \mathcal{K}_2, ..., \mathcal{K}_N$, which are sorted based on their intersection intervals $[a_1, b_1], [a_2, b_2], ..., [a_N, b_N]$.
• Figure 3: Overview of our algorithm. (a) We first use 2D Gaussians to parameterize the geometry field, $F$. These kernels are expected to cluster around the surface. (b) We then convert the geometry field into the density field $\sigma$ and lastly, (c) we leverage our refined volume splatting algorithm for differentiable rendering.
• Figure 4: (a) An illustration of intersecting the $i^\text{th}$ Gaussian surfel, whose normal vector is denoted as $\mathbf{n}_i$ that is perpendicular to the plane containing the surfel, with a ray, whose direction is denoted as $\bm \omega$. The depth of the intersected point is denoted as $t_i$, and the intersected point is denoted as $\mathbf{x}(t_i)$. The value of the Gaussian surfel at $\mathbf{x}(t_i)$ is denoted as $f_i$. (b) We analyze the intersection by creating a 2D coordinate plane which is parallel to the $\mathbf{n}_i$ and $\mathbf{\omega}$, and passes through $\mathbf{x}(t_i)$. The angle between $\mathbf{n}_i$ and $\bm \omega$ is denoted as $\theta_i$. (c) On the 2D coordinate plane, within an infinitesimally small range, the intersected line, drawn as an orange line, between the Gaussian surfel and the plane can be seen as having constant geometry field value $F = f_i-c$. (d) We expand the intersected line by linearly decaying it into $-c$ with length $h$ and direction $\mathbf{n}_i$. It gives the surfel a 3D width, which does not follow the Gaussian distribution. As $h\to0$, it can be seen as equivalent to (c).
• Figure 5: Illustration of opacity value on different Gaussian surfels. (a) 2DGS only allows the opacity to reach $1$ at the center. (b) Due to the enforced transformation from geometry field into density as in \ref{['equation-rho-result-ours']}, the opacity on a Gaussian surfel in our case does not follow a Gaussian distribution, thus differentiating it from Gaussian splatting. Furthermore, we also allow a larger central area to reach $1$, thus making the Gaussian surfel more opaque which benefits the surface reconstruction.

Theorems & Definitions (5)

• proof
• Lemma 1
• Theorem 1
• Theorem 2
• proof