SpiKernel: A Kernel Size Exploration Methodology for Improving Accuracy of the Embedded Spiking Neural Network Systems

TL;DR

SpiKernel addresses the need for high-accuracy spiking neural networks with constrained memory in embedded systems. It introduces a kernel-size exploration methodology that couples kernel-size analysis with NAS-inspired search, while reducing search iterations to $1000$x. Key contributions include (i) evaluating kernel sizes up to $7\times7$ and defining six kernel sets, (ii) a NAS-based SNN generation process with memory-aware model selection, and (iii) empirical validation on CIFAR10/100/TinyImageNet showing improved accuracy and up to $4.8\times$ faster search. The method enables efficient, embedded-ready SNN deployment by balancing accuracy gains with memory budgets through a scalable kernel-size design.

Abstract

Spiking Neural Networks (SNNs) can offer ultra-low power/energy consumption for machine learning-based application tasks due to their sparse spike-based operations. Currently, most of the SNN architectures need a significantly larger model size to achieve higher accuracy, which is not suitable for resource-constrained embedded applications. Therefore, developing SNNs that can achieve high accuracy with acceptable memory footprint is highly needed. Toward this, we propose SpiKernel, a novel methodology that improves the accuracy of SNNs through kernel size exploration. Its key steps include (1) investigating the impact of different kernel sizes on the accuracy, (2) devising new sets of kernel sizes, (3) generating SNN architectures using neural architecture search based on the selected kernel sizes, and (4) analyzing the accuracy-memory trade-offs for SNN model selection. The experimental results show that our SpiKernel achieves higher accuracy than state-of-the-art works (i.e., 93.24% for CIFAR10, 70.84% for CIFAR100, and 62% for TinyImageNet) with less than 10M parameters and up to 4.8x speed-up of searching time, thereby making it suitable for embedded applications.

Figures (7)

• Figure 1: (a) Accuracy and memory footprints of different SNNs for the CIFAR10: AutoSNN Ref_Na_AutoSNN_ICML22, ResNet11 Ref_Lee_SpikeBackprop_FNINS20, ResNet19 Ref_Zheng_LargerSNNs_AAAI21, CIFARNet1 Ref_Wu_DirectTrainSNNs_AAAI19, CIFARNet2 Ref_Fang_MemTConstantSNNs_ICCV21; adapted from the studies in Ref_Na_AutoSNN_ICML22. (b) Experimental results considering the CIFAR100 dataset and two SNN architectures with different sets of kernel sizes: one architecture with 1x1 and 3x3, and another one with 1x1 and 5x5. This shows that a larger kernel size may improve SNN accuracy.
• Figure 2: The SNN macro-architecture, employing 2 neural cells; where a neural cell is defined as a directed acyclic graph with each connection edge represents a specific pre-defined operation; adapted from Ref_Kim_SNASNet_ECCV22.
• Figure 3: (a) Our SpiKernel methodology with novel contributions highlighted in blue boxes. (b) The SNN macro-architecture considered in the SpiKernel. (c) The neural cell in the SpiKernel, showing the enhanced pre-defined operations with additional kernel sizes and operations in 'blue text' as compared to the original ones shown in Fig. \ref{['Fig_MacroArch']}.
• Figure 4: Results of experimental case studies using NAS for the CIFAR100 dataset, while considering different sets of kernel sizes: 1x1Conv and 3x3Conv (i.e., 1x1_3x3_5000x), 1x1Conv and 5x5Conv (i.e., 1x1_5x5_5000x), as well as 1x1Conv and 7x7Conv (i.e., 1x1_7x7_5000x).
• Figure 5: Experimental results for the test accuracy of different SNNs on (a) CIFAR10, (b) CIFAR100, and (c) TinyImageNet datasets.