Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

SpikeGS: Reconstruct 3D scene via fast-moving bio-inspired sensors

Yijia Guo, Liwen Hu, Yuanxi Bai, Jiawei Yao, Lei Ma, Tiejun Huang

TL;DR

SpikeGS tackles the challenge of real-time 3D reconstruction with fast-moving cameras by fusing Bayer-pattern spike streams into the 3D Gaussian Splatting framework. It introduces time accumulation rasterization and interval supervision to extract sharp geometry and texture from high-temporal-resolution, texture-lean spike data captured within one second. The method demonstrates state-of-the-art novel view synthesis on both synthetic and real spike datasets, and exhibits strong robustness to speed variation. This work enables high-speed 3D scene reconstruction without relying on sharp RGB frames, broadening applicability to moving platforms and dynamic environments.

Abstract

3D Gaussian Splatting (3DGS) demonstrates unparalleled superior performance in 3D scene reconstruction. However, 3DGS heavily relies on the sharp images. Fulfilling this requirement can be challenging in real-world scenarios especially when the camera moves fast, which severely limits the application of 3DGS. To address these challenges, we proposed Spike Gausian Splatting (SpikeGS), the first framework that integrates the spike streams into 3DGS pipeline to reconstruct 3D scenes via a fast-moving bio-inspired camera. With accumulation rasterization, interval supervision, and a specially designed pipeline, SpikeGS extracts detailed geometry and texture from high temporal resolution but texture lacking spike stream, reconstructs 3D scenes captured in 1 second. Extensive experiments on multiple synthetic and real-world datasets demonstrate the superiority of SpikeGS compared with existing spike-based and deblur 3D scene reconstruction methods. Codes and data will be released soon.

SpikeGS: Reconstruct 3D scene via fast-moving bio-inspired sensors

TL;DR

SpikeGS tackles the challenge of real-time 3D reconstruction with fast-moving cameras by fusing Bayer-pattern spike streams into the 3D Gaussian Splatting framework. It introduces time accumulation rasterization and interval supervision to extract sharp geometry and texture from high-temporal-resolution, texture-lean spike data captured within one second. The method demonstrates state-of-the-art novel view synthesis on both synthetic and real spike datasets, and exhibits strong robustness to speed variation. This work enables high-speed 3D scene reconstruction without relying on sharp RGB frames, broadening applicability to moving platforms and dynamic environments.

Abstract

3D Gaussian Splatting (3DGS) demonstrates unparalleled superior performance in 3D scene reconstruction. However, 3DGS heavily relies on the sharp images. Fulfilling this requirement can be challenging in real-world scenarios especially when the camera moves fast, which severely limits the application of 3DGS. To address these challenges, we proposed Spike Gausian Splatting (SpikeGS), the first framework that integrates the spike streams into 3DGS pipeline to reconstruct 3D scenes via a fast-moving bio-inspired camera. With accumulation rasterization, interval supervision, and a specially designed pipeline, SpikeGS extracts detailed geometry and texture from high temporal resolution but texture lacking spike stream, reconstructs 3D scenes captured in 1 second. Extensive experiments on multiple synthetic and real-world datasets demonstrate the superiority of SpikeGS compared with existing spike-based and deblur 3D scene reconstruction methods. Codes and data will be released soon.
Paper Structure (20 sections, 16 equations, 8 figures, 4 tables)

This paper contains 20 sections, 16 equations, 8 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Novel View Synthesis on Traditional Cameras
  4. Novel View Synthesis on Bio-inspired Cameras
  5. Datasets
  6. Synthetic SpikeGS Dataset
  7. Real-world SpikeGS Dataset
  8. Method
  9. Preliminary:3D Gaussian Splatting
  10. Color Spike Camera Model
  11. Time Accumulation Rasterization
  12. Mutually Constrained Training and Loss Function
  13. Experiment
  14. Implementation Details
  15. Experimental Settings
  16. ...and 5 more sections

Figures (8)

  • Figure 1: Brief introduction of SpikeGS. Given high temporal resolution but texture lacking spike stream captured by a fast-moving spike camera in one second, our method can reconstruct a sharp radiance field in minutes and achieves clear real-time rendering results.
  • Figure 2: Overview of our SpikeGS. We first reconstruct Bayer-pattern spike streams into spike intervals and spike accumulation (details are shown in Fig. \ref{['tfi_tfp']}). Unlike 3DGS, we adopt spike intervals to initialize SFM points, camera poses, and Gaussian splats. We then embed the time accumulation process into the rasterizer to calibrate the colorization while maintaining multi-view consistency. By progressively optimizing the 3DGS parameters using an accumulation loss and an interval loss, our method facilitates high-quality 3DGS reconstruction.
  • Figure 3: The setup for capturing the real-world dataset, SpikeGS-dataset. First, we fix the toy onto a tray, then the tray is fixed to a motor. Finally, we use a spike camera to capture the high-speed rotating toy. ① (②) controls the sampling of the spike camera (the rotation speed of the motor).
  • Figure 4: Working principle of each pixel in spike camera. Black (white) circle denotes a (no) spike.
  • Figure 5: Left: Overview of Bayer-pattern spike stream reconstruction. We employ a Bayer filter to extract the spike streams of a certain color and calculate accumulating/interval results separately. Right: Reconstruction details. We estimate light intensity by accumulating the number of spikes over a period of time (equation \ref{['eq_TFP1']}) or calculating the interval between adjacent spikes (equation \ref{['eq_TFI1']}).
  • ...and 3 more figures