CADE: Cosine Annealing Differential Evolution for Spiking Neural Network

TL;DR

This work tackles the challenge of training Spiking Neural Networks (SNNs) with non-differentiable spike events by introducing Cosine Annealing Differential Evolution (CADE), which dynamically tunes the mutation factor $F$ and crossover rate $CR$ during optimization of a Spiking Element Wise (SEW) ResNet. CADE employs four CR/$F$ update strategies under a cosine schedule and leverages transfer learning to initialize the population, enhancing diversity and convergence. Empirical results on CIFAR-10/100 and CIFAR-100-C show faster convergence, improved accuracy (up to about $0.52$ percentage points) and stronger robustness compared to gradient-based methods and other DE variants. The approach demonstrates the value of scheduling hyperparameters in DE for SNNs and provides open-source code to facilitate adoption and further research, with implications for energy-efficient neuromorphic computing.

Abstract

Spiking neural networks (SNNs) have gained prominence for their potential in neuromorphic computing and energy-efficient artificial intelligence, yet optimizing them remains a formidable challenge for gradient-based methods due to their discrete, spike-based computation. This paper attempts to tackle the challenges by introducing Cosine Annealing Differential Evolution (CADE), designed to modulate the mutation factor (F) and crossover rate (CR) of differential evolution (DE) for the SNN model, i.e., Spiking Element Wise (SEW) ResNet. Extensive empirical evaluations were conducted to analyze CADE. CADE showed a balance in exploring and exploiting the search space, resulting in accelerated convergence and improved accuracy compared to existing gradient-based and DE-based methods. Moreover, an initialization method based on a transfer learning setting was developed, pretraining on a source dataset (i.e., CIFAR-10) and fine-tuning the target dataset (i.e., CIFAR-100), to improve population diversity. It was found to further enhance CADE for SNN. Remarkably, CADE elevates the performance of the highest accuracy SEW model by an additional 0.52 percentage points, underscoring its effectiveness in fine-tuning and enhancing SNNs. These findings emphasize the pivotal role of a scheduler for F and CR adjustment, especially for DE-based SNN. Source Code on Github: https://github.com/Tank-Jiang/CADE4SNN.

Figures (2)

• Figure 1: Pipeline of the proposed CADE for Spiking Neural Network.
• Figure 2: SEW residual block