Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Towards Efficient and Accurate Spiking Neural Networks via Adaptive Bit Allocation

Xingting Yao, Qinghao Hu, Fei Zhou, Tielong Liu, Gang Li, Peisong Wang, Jian Cheng

TL;DR

To address memory and computation blowups in multi-bit SNNs, the paper introduces adaptive bit allocation with learnable layer-wise temporal lengths and bit widths for spikes and weights, along with a refined spiking neuron and a step-size renewal mechanism. The approach enables fine-grained resource allocation, reduces Bit Budget and S-ACE while improving accuracy on CIFAR, CIFAR-DVS, DVS-GESTURE, and ImageNet, with notable gains on SEW-ResNet34. The contributions include gradient-based learnable bit widths, a temporally aware refined neuron, a theoretically grounded step-size mismatch analysis and renewal, and comprehensive ablations and hardware-oriented discussions. The work provides open-source code and demonstrates practical impact for energy-efficient neuromorphic computing.

Abstract

Multi-bit spiking neural networks (SNNs) have recently become a heated research spot, pursuing energy-efficient and high-accurate AI. However, with more bits involved, the associated memory and computation demands escalate to the point where the performance improvements become disproportionate. Based on the insight that different layers demonstrate different importance and extra bits could be wasted and interfering, this paper presents an adaptive bit allocation strategy for direct-trained SNNs, achieving fine-grained layer-wise allocation of memory and computation resources. Thus, SNN's efficiency and accuracy can be improved. Specifically, we parametrize the temporal lengths and the bit widths of weights and spikes, and make them learnable and controllable through gradients. To address the challenges caused by changeable bit widths and temporal lengths, we propose the refined spiking neuron, which can handle different temporal lengths, enable the derivation of gradients for temporal lengths, and suit spike quantization better. In addition, we theoretically formulate the step-size mismatch problem of learnable bit widths, which may incur severe quantization errors to SNN, and accordingly propose the step-size renewal mechanism to alleviate this issue. Experiments on various datasets, including the static CIFAR and ImageNet datasets and the dynamic CIFAR-DVS and DVS-GESTURE datasets, demonstrate that our methods can reduce the overall memory and computation cost while achieving higher accuracy. Particularly, our SEWResNet-34 can achieve a 2.69\% accuracy gain and 4.16$\times$ lower bit budgets over the advanced baseline work on ImageNet. This work is open-sourced at \href{https://github.com/Ikarosy/Towards-Efficient-and-Accurate-Spiking-Neural-Networks-via-Adaptive-Bit-Allocation}{this link}.

Towards Efficient and Accurate Spiking Neural Networks via Adaptive Bit Allocation

TL;DR

To address memory and computation blowups in multi-bit SNNs, the paper introduces adaptive bit allocation with learnable layer-wise temporal lengths and bit widths for spikes and weights, along with a refined spiking neuron and a step-size renewal mechanism. The approach enables fine-grained resource allocation, reduces Bit Budget and S-ACE while improving accuracy on CIFAR, CIFAR-DVS, DVS-GESTURE, and ImageNet, with notable gains on SEW-ResNet34. The contributions include gradient-based learnable bit widths, a temporally aware refined neuron, a theoretically grounded step-size mismatch analysis and renewal, and comprehensive ablations and hardware-oriented discussions. The work provides open-source code and demonstrates practical impact for energy-efficient neuromorphic computing.

Abstract

Multi-bit spiking neural networks (SNNs) have recently become a heated research spot, pursuing energy-efficient and high-accurate AI. However, with more bits involved, the associated memory and computation demands escalate to the point where the performance improvements become disproportionate. Based on the insight that different layers demonstrate different importance and extra bits could be wasted and interfering, this paper presents an adaptive bit allocation strategy for direct-trained SNNs, achieving fine-grained layer-wise allocation of memory and computation resources. Thus, SNN's efficiency and accuracy can be improved. Specifically, we parametrize the temporal lengths and the bit widths of weights and spikes, and make them learnable and controllable through gradients. To address the challenges caused by changeable bit widths and temporal lengths, we propose the refined spiking neuron, which can handle different temporal lengths, enable the derivation of gradients for temporal lengths, and suit spike quantization better. In addition, we theoretically formulate the step-size mismatch problem of learnable bit widths, which may incur severe quantization errors to SNN, and accordingly propose the step-size renewal mechanism to alleviate this issue. Experiments on various datasets, including the static CIFAR and ImageNet datasets and the dynamic CIFAR-DVS and DVS-GESTURE datasets, demonstrate that our methods can reduce the overall memory and computation cost while achieving higher accuracy. Particularly, our SEWResNet-34 can achieve a 2.69\% accuracy gain and 4.16 lower bit budgets over the advanced baseline work on ImageNet. This work is open-sourced at \href{https://github.com/Ikarosy/Towards-Efficient-and-Accurate-Spiking-Neural-Networks-via-Adaptive-Bit-Allocation}{this link}.

Paper Structure

This paper contains 54 sections, 2 theorems, 29 equations, 7 figures, 11 tables, 3 algorithms.

Table of Contents

  1. Introduction
  2. Related work
  3. Supervised direct learning of SNNs.
  4. Multi-bit spiking neural networks.
  5. Mixed-precision quantization.
  6. Preliminaries
  7. Multi-bit spiking neuron
  8. Quantization of weight parameters.
  9. Legitimate computation and memory measurement.
  10. Methodology
  11. Parametrization
  12. Formulation of the refined parametric spiking neuron
  13. Adaptive bit width
  14. Gradient calculations
  15. Loss function
  16. ...and 39 more sections

Key Result

Theorem 1

In the quantization process of the variable $x$: $x_q = s\cdot clip(\left\lfloor\frac{x}{s}\right\rceil,0,2^b-1)$, where $x\sim 2N(0,\sigma^2)| x>0$; $b$ and $s$ respectively denote the bit width and the quantization step size. Let the quantization step size $s$ be the statistically optimal: $3\sig

Figures (7)

  • Figure 1: Comparisons with other advanced direct-trained SNNs using ResNet-based architectures on CIFAR100 and ImageNet-1k. Our models maintain the same level of model size as the baselines.
  • Figure 2: Overview of the proposed bit allocation method. Green notations denote the parametrized constants. (a) Adaptive-bit-allocation training pipline, where the bit widths $T_l$, $B_{s,l}$, and $B_{w,l}$ are made learnable and adaptive. $T_l$ is matched inter-layer and intra-layer to ensure fluent dataflow. The step-size renewal mechanism is also proposed to alleviate the step-size mismatch issue. (b) depicts the neuron formulation (bottom) and the potential division (top) of the previous spiking neuron (left) and our bit-adaptive spiking neuron (right). The temporal length $T_l$ is layer-wise learnable and a shift $V^2_{th,l}$ is added to the potential threshold $V^1_{th,l}$.
  • Figure 3: Average bit width changes of ResNet-20 and Spikformer on CIFAR-10 during the adaptive-bit-width training. Tar. abbreviates target bit width. W and S denote weight and spike bit width, respectively.
  • Figure 4: Statistics of each layer's bit widths. Since layer #1 has two time-steps, we report the averaged spike bit width for clarity.
  • Figure 5: Bit-serial style neural integration unit that can handle various W/S/T combinations. Psum represents the input current. "Reg." denotes register. "Div." is divider. For simplicity, we set S=2 and T=2.
  • ...and 2 more figures

Theorems & Definitions (2)

  • Theorem 1
  • Lemma 1