Bridge the Gap between SNN and ANN for Image Restoration

TL;DR

The paper addresses the energy cost of ANN-based image restoration and the slow training of SNNs by introducing SpikerIR, a lightweight SNN trained via asymmetric ANN-SNN distillation (H-KD) that leverages ANN decoder features as prompts. The approach combines a compact encoder-decoder SNN with MSE, L1, and FFT losses to align with ANN representations, using a hyperparameterized KD schedule to accelerate learning. Across five degradation tasks and multiple teacher ANNs, SpikerIR achieves competitive restoration quality with only a tiny fraction of the parameters and energy, demonstrating a viable path to edge-friendly, high-performance image restoration. The work highlights both substantial energy efficiency gains and remaining challenges in inference latency and hardware optimization, pointing to practical deployment opportunities and future refinements.

Abstract

Models of dense prediction based on traditional Artificial Neural Networks (ANNs) require a lot of energy, especially for image restoration tasks. Currently, neural networks based on the SNN (Spiking Neural Network) framework are beginning to make their mark in the field of image restoration, especially as they typically use less than 10\% of the energy of ANNs with the same architecture. However, training an SNN is much more expensive than training an ANN, due to the use of the heuristic gradient descent strategy. In other words, the process of SNN's potential membrane signal changing from sparse to dense is very slow, which affects the convergence of the whole model.To tackle this problem, we propose a novel distillation technique, called asymmetric framework (ANN-SNN) distillation, in which the teacher is an ANN and the student is an SNN. Specifically, we leverage the intermediate features (feature maps) learned by the ANN as hints to guide the training process of the SNN. This approach not only accelerates the convergence of the SNN but also improves its final performance, effectively bridging the gap between the efficiency of the SNN and the superior learning capabilities of ANN. Extensive experimental results show that our designed SNN-based image restoration model, which has only 1/300 the number of parameters of the teacher network and 1/50 the energy consumption of the teacher network, is as good as the teacher network in some denoising tasks.

Figures (10)

• Figure 1: The Coxcomb of visual results and key evaluation metrics. Our SNN-based method transfers the knowledge from the ANNs for better image restoration performance.
• Figure 2: Overview of our method. SpikerIR transfers the knowledge from pre-trained ANNs' decoders to enhance the comprehension of degradation images, helping output high-quality content features. SpikerIR is designed as an encoder-decoder architecture, which mainly contains the Spiking Block with Spike Convolution Unit and Multi-dimensional Attention.
• Figure 3: Visual results on Image denoising. Top row: the noise level is $15$. Middle row: the noise level is $25$. Bottom row: the noise level is $50$.
• Figure 4: Single image deraining. Compared to the teacher model, our SpikerIR achieves comparable performance with significantly fewer parameters.
• Figure 5: Visual results on image deblurring. Top row: Realworld deblurring on RealBlur dataset. Bottom row: Synthetic deblurring on Gopro Dataset.