SCSim: A Realistic Spike Cameras Simulator

TL;DR

SCSim tackles the lack of realistic spike-stream datasets by introducing a spike-camera simulator with a circuit-aware noise model and autonomous driving scene generation. It combines a detailed spike camera model with temporal and spatial noise to produce authentic spike streams, validated against real data and used to improve reconstruction through RHDD and WGSE finetuning. A spike-based noise evaluation framework (SNEE) enables calibration against measurements, and auxiliary Rand Scenes/Label Generation functions streamline large-scale dataset creation. The work demonstrates that SCSim yields more realistic spike streams than prior simulators and can significantly advance spike-camera tasks such as reconstruction. The project is publicly available at the associated GitHub repository.

Abstract

Spike cameras, with their exceptional temporal resolution, are revolutionizing high-speed visual applications. Large-scale synthetic datasets have significantly accelerated the development of these cameras, particularly in reconstruction and optical flow. However, current synthetic datasets for spike cameras lack sophistication. Addressing this gap, we introduce SCSim, a novel and more realistic spike camera simulator with a comprehensive noise model. SCSim is adept at autonomously generating driving scenarios and synthesizing corresponding spike streams. To enhance the fidelity of these streams, we've developed a comprehensive noise model tailored to the unique circuitry of spike cameras. Our evaluations demonstrate that SCSim outperforms existing simulation methods in generating authentic spike streams. Crucially, SCSim simplifies the creation of datasets, thereby greatly advancing spike-based visual tasks like reconstruction. Our project refers to https://github.com/Acnext/SCSim.

Figures (8)

• Figure 1: An example of the output of proposed simulator, SCSim, in a rand scene.
• Figure 2: Framework of spike camera simulator, SCSim. First, according to graphic scenes and camera parameters, image sequences, and vision task labels are generated by render backends. Then we convert the image sequences into spike streams based on our spike camera model. The blue (red) boxes denote the auxiliary functions (the spike camera model) in SCSim.
• Figure 3: The principle and circuit of spike camera. (a-c) is the working principle of the accumulator, reset model, and check mechanism in a pixel circuit of spike camera. "Star" means that we have considered the corresponding noise.
• Figure 4: The difference of rand scenes function between SPCS2022scflow and our SCSim. (a) Rand scenes function in SPCS where rand objects move and the background is a static HDR image. (b) Rand scenes function in SCSim. By combining rand cars, street information, and city models, it can randomly generate automatic driving scenes.
• Figure 5: The spike camera shoots a computer monitor under different grayscale backgrounds.