Gaussian Set Surface Reconstruction through Per-Gaussian Optimization

TL;DR

Gaussian Set Surface Reconstruction (GSSR) is proposed, a method designed to distribute Gaussians evenly along the latent surface while aligning their dominant normals with the surface normal, enabling intuitive scene editing and efficient generation of novel Gaussian-based 3D environments.

Abstract

3D Gaussian Splatting (3DGS) effectively synthesizes novel views through its flexible representation, yet fails to accurately reconstruct scene geometry. While modern variants like PGSR introduce additional losses to ensure proper depth and normal maps through Gaussian fusion, they still neglect individual placement optimization. This results in unevenly distributed Gaussians that deviate from the latent surface, complicating both reconstruction refinement and scene editing. Motivated by pioneering work on Point Set Surfaces, we propose Gaussian Set Surface Reconstruction (GSSR), a method designed to distribute Gaussians evenly along the latent surface while aligning their dominant normals with the surface normal. GSSR enforces fine-grained geometric alignment through a combination of pixel-level and Gaussian-level single-view normal consistency and multi-view photometric consistency, optimizing both local and global perspectives. To further refine the representation, we introduce an opacity regularization loss to eliminate redundant Gaussians and apply periodic depth- and normal-guided Gaussian reinitialization for a cleaner, more uniform spatial distribution. Our reconstruction results demonstrate significantly improved geometric precision in Gaussian placement, enabling intuitive scene editing and efficient generation of novel Gaussian-based 3D environments. Extensive experiments validate GSSR's effectiveness, showing enhanced geometric accuracy while preserving high-quality rendering performance.

Figures (7)

• Figure 1: Comparison of Gaussian primitives distributions from 2DGS, PGSR, and our method across two scenes. Gaussian centers are visualized as colored points, with hue indicating surface distance error. Areas where the white background is visible through the Gaussians denote incomplete surface coverage. Red and green boxes highlights regions with improved accuracy and coverage, respectively. Quantitative and visual results confirm that our approach achieves both a lower reconstruction error and a more complete Gaussian representation.
• Figure 2: Overview of the proposed GSSR pipeline. Given multi-view posed images and initialized 3D Gaussians, we render the depth map, depth-based normals and alpha-blended normals. The optimization stage includes four major components: (1) single-view normal consistency, (2) multi-view photometric consistency (3) RGB rendering loss, and (4) opacity regularization. Additionally, Gaussians are periodically resampled using our view-based opacity-guided strategy, resulting in a more uniform and accurate distribution.
• Figure 3: Illustration of (a) $L_{normal}$ and (b) $L_{normal-G}$. Gaussian Normal is derived from the direction of the Gaussian's minimum scale axis. Rendered Normal is computed via alpha blending of Gaussian normals along each pixel ray. Depth-Based Normal is estimated from the depth gradients of neighboring pixels.
• Figure 4: Illustration of pixel-level (a) and Gaussian-level (b) multi-view photometric loss.
• Figure 5: Comparison of Gaussian density distribution between PGSR and our method. Our approach produces lower Gaussian density, especially on planar regions (see red arrows), where the splats are more compact and narrowly distributed, demonstrating better geometric compactness and representation efficiency.