Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

LitE-SNN: Designing Lightweight and Efficient Spiking Neural Network through Spatial-Temporal Compressive Network Search and Joint Optimization

Qianhui Liu, Jiaqi Yan, Malu Zhang, Gang Pan, Haizhou Li

TL;DR

This work addresses the challenge of deploying accurate SNNs on resource-constrained edge devices by integrating both spatial and temporal compression into neural architecture search. It introduces Compressive Convolution (CompConv) to enable pruning and mixed-precision quantization with shared weights, and a compressive timestep search to automatically select timesteps under cost constraints, all within a multi-objective joint optimization framework. The approach achieves competitive or superior accuracy on CIFAR-10, CIFAR-100, and Google Speech Command while drastically reducing model size and Bit-SynOps, and demonstrates favorable energy characteristics relative to ANN baselines and prior SNN methods. The findings highlight the practical potential of jointly optimizing architecture and spatio-temporal compression to enable efficient, real-world SNN deployments on edge hardware and neuromorphic platforms.

Abstract

Spiking Neural Networks (SNNs) mimic the information-processing mechanisms of the human brain and are highly energy-efficient, making them well-suited for low-power edge devices. However, the pursuit of accuracy in current studies leads to large, long-timestep SNNs, conflicting with the resource constraints of these devices. In order to design lightweight and efficient SNNs, we propose a new approach named LitE-SNN that incorporates both spatial and temporal compression into the automated network design process. Spatially, we present a novel Compressive Convolution block (CompConv) to expand the search space to support pruning and mixed-precision quantization. Temporally, we are the first to propose a compressive timestep search to identify the optimal number of timesteps under specific computation cost constraints. Finally, we formulate a joint optimization to simultaneously learn the architecture parameters and spatial-temporal compression strategies to achieve high performance while minimizing memory and computation costs. Experimental results on CIFAR-10, CIFAR-100, and Google Speech Command datasets demonstrate our proposed LitE-SNNs can achieve competitive or even higher accuracy with remarkably smaller model sizes and fewer computation costs.

LitE-SNN: Designing Lightweight and Efficient Spiking Neural Network through Spatial-Temporal Compressive Network Search and Joint Optimization

TL;DR

This work addresses the challenge of deploying accurate SNNs on resource-constrained edge devices by integrating both spatial and temporal compression into neural architecture search. It introduces Compressive Convolution (CompConv) to enable pruning and mixed-precision quantization with shared weights, and a compressive timestep search to automatically select timesteps under cost constraints, all within a multi-objective joint optimization framework. The approach achieves competitive or superior accuracy on CIFAR-10, CIFAR-100, and Google Speech Command while drastically reducing model size and Bit-SynOps, and demonstrates favorable energy characteristics relative to ANN baselines and prior SNN methods. The findings highlight the practical potential of jointly optimizing architecture and spatio-temporal compression to enable efficient, real-world SNN deployments on edge hardware and neuromorphic platforms.

Abstract

Spiking Neural Networks (SNNs) mimic the information-processing mechanisms of the human brain and are highly energy-efficient, making them well-suited for low-power edge devices. However, the pursuit of accuracy in current studies leads to large, long-timestep SNNs, conflicting with the resource constraints of these devices. In order to design lightweight and efficient SNNs, we propose a new approach named LitE-SNN that incorporates both spatial and temporal compression into the automated network design process. Spatially, we present a novel Compressive Convolution block (CompConv) to expand the search space to support pruning and mixed-precision quantization. Temporally, we are the first to propose a compressive timestep search to identify the optimal number of timesteps under specific computation cost constraints. Finally, we formulate a joint optimization to simultaneously learn the architecture parameters and spatial-temporal compression strategies to achieve high performance while minimizing memory and computation costs. Experimental results on CIFAR-10, CIFAR-100, and Google Speech Command datasets demonstrate our proposed LitE-SNNs can achieve competitive or even higher accuracy with remarkably smaller model sizes and fewer computation costs.
Paper Structure (16 sections, 20 equations, 3 figures, 5 tables)

This paper contains 16 sections, 20 equations, 3 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Neural Architecture Search
  4. Model Compression
  5. Method
  6. Preliminary
  7. Compressive Convolution (CompConv)
  8. Compressive Timestep Search
  9. Loss Function and Joint Optimization Algorithm
  10. Experiments
  11. Experimental Settings
  12. Comparison with Existing SNNs
  13. Effectiveness of $\lambda_1$ and $\lambda_2$
  14. Comparison with Sequential Optimization
  15. Energy Consumption
  16. ...and 1 more sections

Figures (3)

  • Figure 1: Hierarchical search space. (a) Cell structure search space with 3 nodes. (b) Within one node, two candidate operations are summed filtered by a sign function.
  • Figure 2: (a) A naive solution that integrates mixed-quantization with pruning. (b) Our proposed CompConv for a more efficient integration. (c) Visualization of the distribution of $W$, $\bar{e}(W)$, and $W_{output}$ with pruning rates. (d) Our proposed timestep search solution.
  • Figure 3: Ablation study of $\lambda_1$ and $\lambda_2$.