Learning A Spiking Neural Network for Efficient Image Deraining

TL;DR

This paper tackles single-image deraining under energy constraints by leveraging spiking neural networks (SNNs). It introduces ESDNet, an encoder–decoder with a Spiking Residual Block (SRB) that uses a Spike Convolution Unit and a Mixed Attention Unit, plus a Feature Refinement Block to convert spike features back to image representations; training employs a gradient surrogate rather than ANN–SNN conversion. The approach yields competitive PSNR/SSIM compared to ANN-based methods while achieving substantial energy savings, evidenced by experiments across synthetic and real rain datasets. The work demonstrates the practicality of direct-trained SNNs for pixel-level restoration tasks and provides code for reproducibility.

Abstract

Recently, spiking neural networks (SNNs) have demonstrated substantial potential in computer vision tasks. In this paper, we present an Efficient Spiking Deraining Network, called ESDNet. Our work is motivated by the observation that rain pixel values will lead to a more pronounced intensity of spike signals in SNNs. However, directly applying deep SNNs to image deraining task still remains a significant challenge. This is attributed to the information loss and training difficulties that arise from discrete binary activation and complex spatio-temporal dynamics. To this end, we develop a spiking residual block to convert the input into spike signals, then adaptively optimize the membrane potential by introducing attention weights to adjust spike responses in a data-driven manner, alleviating information loss caused by discrete binary activation. By this way, our ESDNet can effectively detect and analyze the characteristics of rain streaks by learning their fluctuations. This also enables better guidance for the deraining process and facilitates high-quality image reconstruction. Instead of relying on the ANN-SNN conversion strategy, we introduce a gradient proxy strategy to directly train the model for overcoming the challenge of training. Experimental results show that our approach gains comparable performance against ANN-based methods while reducing energy consumption by 54%. The code source is available at https://github.com/MingTian99/ESDNet.

Figures (5)

• Figure 1: Model parameters and performance comparison between our proposed ESDNet and other deraining methods on the Rain200L dataset. The circle sizes represent the FLOPs of the various methods. The results show that the proposed method obtains a superior balance between model complexity and deraining performance.
• Figure 2: Visualization of image patch. From the pixel value map, it can be seen that the rain patch is generally higher than the surrounding areas without rain. The neuron voltage map represents the voltage heatmap learned by the LIF model. For the rain patches, the voltage changes of neurons are more drastic. This indicates that neurons are in an active state with a higher spike rate.
• Figure 3: The architecture of the proposed Efficient Spiking Deraining Network (ESDNet), which takes input rainy images and generates output derained images. It mainly contains (1) the Spiking Residual Block (SRB) with Spike Convolution Unit (SCU) and Mixed Attention Unit (MAU), (2) Feature Refinement Block (FRB). $\operatorname{tdBN}$ refers to threshold-dependent batch normalization.
• Figure 4: Visual comparison of the Rain200H dataset. Best viewed by zooming in on the figures on high-resolution displays.
• Figure 5: Visual comparison on the RW-Data. Best viewed by zooming in the figures on high-resolution displays.