High-speed and High-quality Vision Reconstruction of Spike Camera with Spike Stability Theorem

TL;DR

This work tackles real-time high-quality texture reconstruction from spike-camera data by introducing the Spike Stability Theorem, which links spike stream dynamics to stable light intensity under a fixed accumulation rate $Q_0$. Building on this theory, it proposes two parameter-free reconstruction algorithms, First-order Stability Reconstruction (FSR) and Second-order Stability Reconstruction (SSR), where light intensity is estimated from mean inter-spike intervals as $I=\frac{255}{\frac{1}{N}\sum_i T_0(i)}$, and segmentation into absolutely stable sequences enables robust reconstruction. The methods are FPGA-friendly and demonstrated on PKU-Spike-High-Speed and SpikeCityPCL datasets, achieving a best trade-off between reconstruction quality and speed and reaching 20,000 FPS on hardware. Overall, the paper provides both theoretical guarantees and practical, real-time edge-end processing foundations for spike-camera vision, with potential extensions to higher-level tasks.

Abstract

Neuromorphic vision sensors, such as the dynamic vision sensor (DVS) and spike camera, have gained increasing attention in recent years. The spike camera can detect fine textures by mimicking the fovea in the human visual system, and output a high-frequency spike stream. Real-time high-quality vision reconstruction from the spike stream can build a bridge to high-level vision task applications of the spike camera. To realize high-speed and high-quality vision reconstruction of the spike camera, we propose a new spike stability theorem that reveals the relationship between spike stream characteristics and stable light intensity. Based on the spike stability theorem, two parameter-free algorithms are designed for the real-time vision reconstruction of the spike camera. To demonstrate the performances of our algorithms, two datasets (a public dataset PKU-Spike-High-Speed and a newly constructed dataset SpikeCityPCL) are used to compare the reconstruction quality and speed of various reconstruction methods. Experimental results show that, compared with the current state-of-the-art (SOTA) reconstruction methods, our reconstruction methods obtain the best tradeoff between the reconstruction quality and speed. Additionally, we design the FPGA implementation method of our algorithms to realize the real-time (running at 20,000 FPS) visual reconstruction. Our work provides new theorem and algorithm foundations for the real-time edge-end vision processing of the spike camera.

Figures (6)

• Figure 1: The interval sequences needed to calculate when segmenting a spike stream $S_0$ by n-order stable judgment. Red dotted circle indicates that the interval sequence $S_n^m$ is empty. The '-' indicates selecting the smaller value as the firing value, and the '+' indicates selecting the larger value as the firing value. An example of spike stream $S_0$ segmented by FSR (First-order Stability Reconstruction) and SSR (Second-order Stability Reconstruction).
• Figure 2: Visual representation of the simplified workflow of FSR (First-order Stability Reconstruction) and SSR (Second-order Stability Reconstruction).
• Figure 3: Overview of FPGA-implemented reconstruction.
• Figure 4: The detail of FPGA-implemented stability module (FSR).
• Figure 5: Images reconstructed by different algorithms under some typical high-speed moving and rich-texture scenes. (rotation, car, and train are from dataset PKU-Spike-High-Speed, and road and building are from dataset SpikeCityPCL)

Theorems & Definitions (8)

• Definition 1
• Definition 2
• Theorem 1
• Lemma 1
• Lemma 2
• proof
• proof
• proof