Learning to Robustly Reconstruct Low-light Dynamic Scenes from Spike Streams

TL;DR

This work tackles robust reconstruction of high-speed scenes from spike camera spike streams under low illumination, where information is sparse. It introduces a bidirectional recurrent reconstruction framework featuring a light-robust representation (LR-Rep) and a fusion module to exploit forward and backward temporal information, including a GISI-based transformation. To evaluate performance, the authors synthesize realistic low-light datasets (RLLR and LLR) and demonstrate that their approach achieves higher PSNR/SSIM and better temporal consistency than prior methods on both synthetic and real spike streams. The proposed method offers a practical solution for reliable spike-camera reconstruction in challenging lighting, with datasets and code to be released after publication.

Abstract

As a neuromorphic sensor with high temporal resolution, spike camera can generate continuous binary spike streams to capture per-pixel light intensity. We can use reconstruction methods to restore scene details in high-speed scenarios. However, due to limited information in spike streams, low-light scenes are difficult to effectively reconstruct. In this paper, we propose a bidirectional recurrent-based reconstruction framework, including a Light-Robust Representation (LR-Rep) and a fusion module, to better handle such extreme conditions. LR-Rep is designed to aggregate temporal information in spike streams, and a fusion module is utilized to extract temporal features. Additionally, we have developed a reconstruction benchmark for high-speed low-light scenes. Light sources in the scenes are carefully aligned to real-world conditions. Experimental results demonstrate the superiority of our method, which also generalizes well to real spike streams. Related codes and proposed datasets will be released after publication.

Figures (12)

• Figure 1: Overview of reconstruction for high-speed spike streams. Left: with decreasing light intensity, more sparse spike streams are difficult to extract features. A black circle is a spike. Middle: (a) The state-of-the-art method, WGSE rec6. The arrow with a gradient color is the timeline. (b) Our reconstruction method. Green (red) lines denote the forward (backward) data flow. ① (②) is the release time of spikes (temporal features). ① (②) in forward and backward data flow is independent. Right: reconstructed results from WGSE and our method.
• Figure 2: Proposed datasets, RLLR and LLR. RLLR includes random scenes and LLR includes carefully designed scenes. A Spike Frame is a slice of generated spike streams on a temporal axis.
• Figure 3: Illustration of the proposed bidirectional recurrent-based reconstruction framework. It includes a light-robust representation, feature extractor (ResNet), fusion, and reconstruction. The green and red lines represent the forward and backward data flow. The two kinds of data flow are independent.
• Figure 4: Illustration of the proposed light-robust representation. We use convolution blocks to extract shallow features from input spike stream and GISI, respectively. Then they are fused by an attention block.
• Figure 5: Illustration of GISI transform for backward in a pixel. (a). Calculate the local inter-spike interval, $\mathbf{LISI}_{t_i}$ from the input spike stream rec5zhao2022learning. (b). Update global inter-spike interval, $\mathbf{GISI}_{t_i}$ based on the release time of backward spike, $\mathbf{Spike}_{t_{i+1}}^b$ and $\mathbf{LISI}_{t_i}$. (c). Maintain and transmit the release time of backward spike, $\mathbf{Spike}_{t_{i}}^b$. Black (white) circle is a (no) spike and the red line is backward data flow.