Spiking GS: Towards High-Accuracy and Low-Cost Surface Reconstruction via Spiking Neuron-based Gaussian Splatting

TL;DR

Spiking GS is proposed to reduce two types of LOPs by integrating spiking neurons into the Gaussian Splatting pipeline by introducing global and local full-precision integrate-and-fire spiking neurons to the opacity and representation function of flattened 3D Gaussians, respectively.

Abstract

3D Gaussian Splatting is capable of reconstructing 3D scenes in minutes. Despite recent advances in improving surface reconstruction accuracy, the reconstructed results still exhibit bias and suffer from inefficiency in storage and training. This paper provides a different observation on the cause of the inefficiency and the reconstruction bias, which is attributed to the integration of the low-opacity parts (LOPs) of the generated Gaussians. We show that LOPs consist of Gaussians with overall low-opacity (LOGs) and the low-opacity tails (LOTs) of Gaussians. We propose Spiking GS to reduce such two types of LOPs by integrating spiking neurons into the Gaussian Splatting pipeline. Specifically, we introduce global and local full-precision integrate-and-fire spiking neurons to the opacity and representation function of flattened 3D Gaussians, respectively. Furthermore, we enhance the density control strategy with spiking neurons' thresholds and a new criterion on the scale of Gaussians. Our method can represent more accurate reconstructed surfaces at a lower cost. The supplementary material and code are available at https://github.com/zju-bmi-lab/SpikingGS.

Figures (9)

• Figure 1: Left: we show a visual comparison of the extracted mesh for the Mic object from the NeRF-Synthetic dataset mildenhall2021nerf. We additionally compare the Gaussians' number (#G), number of Low-Opacity Gaussians (#LOGs), Chamfer Distance (CD), and optimization time between our and previous methods. We regard opacity lower than 0.2 as low-opacity for statistics. Right: we show statics of different methods on the NeRF-Synthetic dataset mildenhall2021nerf in terms of CD, #G, and training time visualized by the size of the circles, bigger circles indicate faster speed; numbers indicate the specific time cost in minutes.
• Figure 2: Illustration of LOPs (LOGs and LOTs). From top to bottom, the first (second) row shows the front (side) view of the Gaussians around the surface.
• Figure 3: Visualization of the side view of the optimized Gaussian around the surface of interest under different situations: ① optimal (according to dai2024highguedon2024sugar), ②③ suboptimal, and ④ ours. The bias of the reconstructed surface can be observed by the distance between the red and green lines.
• Figure 4: Visualization of $\omega$ value on a view ray of (a) Drums and (b) Mic from NeRF-Synthetic Datasetmildenhall2021nerf. We select representative patterns of $\omega$ near the actual surface. The height of each bin represents the $\omega$ value, the red and green lines represent the GT depth and the estimated depth. Short bins indicate the presence of LOPs spreading out around the actual surface positions. Numbers indicate the corresponding depth errors. Statistics on two types of LOPs on (c) Drums, (d) Mic from NeRF-Synthetic Dataset, including the number of Gaussians, the proportion of LOGs, and the number of LOTs (averaged overlaps of Gaussian’s tails with a Gaussian representation function value below 0.1 in multiple views).
• Figure 5: The pipeline of the proposed Spiking GS. Two types of FIF neurons are applied to the flattened Gaussians. A modified density control process can more adaptively adjust the number of Gaussians. Attributes of the Gaussians are updated by photometric loss $\mathcal{L}_{c}$, total variance loss $\mathcal{L}_{t}$, depth distortion loss $\mathcal{L}_{d}$, normal loss $\mathcal{L}_{N}$, scale loss $\mathcal{L}_{s}$, and threshold loss $\mathcal{L}_\alpha$, $\mathcal{L}_\mathrm{p}$.