SpikeDerain: Unveiling Clear Videos from Rainy Sequences Using Color Spike Streams

TL;DR

SpikeDerain presents a single-modal deraining framework that uses color spike streams from a spike camera to recover rain-free videos with high fidelity. It combines a color spike reconstruction pipeline (CSR, SNE, CCP) with spiking residual blocks (SCU, MAU) to jointly reconstruct backgrounds and remove rain, guided by a physically interpretable rain streak synthesis model for synthetic data. The method achieves state-of-the-art PSNR/SSIM across Rain100C, RainSynLight25/Heavy25, and NTURain, and demonstrates robustness under extreme rainfall while enabling energy-efficient neuromorphic computation. This work advances rain deraining by leveraging the continuous, absolute-brightness, color-preserving properties of spike cameras, reducing cross-modal alignment issues, and providing a practical dataset framework for training and evaluation.

Abstract

Restoring clear frames from rainy videos presents a significant challenge due to the rapid motion of rain streaks. Traditional frame-based visual sensors, which capture scene content synchronously, struggle to capture the fast-moving details of rain accurately. In recent years, neuromorphic sensors have introduced a new paradigm for dynamic scene perception, offering microsecond temporal resolution and high dynamic range. However, existing multimodal methods that fuse event streams with RGB images face difficulties in handling the complex spatiotemporal interference of raindrops in real scenes, primarily due to hardware synchronization errors and computational redundancy. In this paper, we propose a Color Spike Stream Deraining Network (SpikeDerain), capable of reconstructing spike streams of dynamic scenes and accurately removing rain streaks. To address the challenges of data scarcity in real continuous rainfall scenes, we design a physically interpretable rain streak synthesis model that generates parameterized continuous rain patterns based on arbitrary background images. Experimental results demonstrate that the network, trained with this synthetic data, remains highly robust even under extreme rainfall conditions. These findings highlight the effectiveness and robustness of our method across varying rainfall levels and datasets, setting new standards for video deraining tasks. The code will be released soon.

Figures (13)

• Figure 1: Comparison of Information Capture between Traditional Video Frames and Spike Streams. Traditional cameras, due to fixed exposure time and sampling frequency, introduce dynamic blur and inter-frame intervals, which degrade the image quality for subsequent tasks. In contrast, spike streams, with their ultra-high temporal resolution, capture high-speed information more effectively.
• Figure 1: Visual comparison between existing datasets and our method for generating rain streaks. There is a phenomenon of discontinuous rain streaks in the existing dataset, while the rain streaks generated by this method are visually more natural and continuous.
• Figure 2: Difference between Event-based and Our Spike-based Rainy Removal. Event cameras only record relative light intensity. In contrast, spike cameras record absolute light intensity, thereby preserving essential static information.
• Figure 2: Qualitative results on Rain100C. Zoom-in for better visualization.
• Figure 3: Architecture of Proposed Spike-Driven Deraining Network (SpikeDerain). Our method leverages color spike streams from a spike camera by integrating a color spike reconstruction module with a multi-stage SNN-based deraining module to efficiently restore rain-free videos.