Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Noise Adaptor: Enhancing Low-Latency Spiking Neural Networks through Noise-Injected Low-Bit ANN Conversion

Chen Li, Bipin. Rajendran

TL;DR

Noise Adaptor is presented, a novel method for constructing competitive low-latency spiking neural networks (SNNs) by converting noise-injected, low-bit artificial neural networks (ANNs) by converting noise-injected, low-bit artificial neural networks (ANNs).

Abstract

We present Noise Adaptor, a novel method for constructing competitive low-latency spiking neural networks (SNNs) by converting noise-injected, low-bit artificial neural networks (ANNs). This approach builds on existing ANN-to-SNN conversion techniques but offers several key improvements: (1) By injecting noise during quantized ANN training, Noise Adaptor better accounts for the dynamic differences between ANNs and SNNs, significantly enhancing SNN accuracy. (2) Unlike previous methods, Noise Adaptor does not require the application of run-time noise correction techniques in SNNs, thereby avoiding modifications to the spiking neuron model and control flow during inference. (3) Our method extends the capability of handling deeper architectures, achieving successful conversions of activation-quantized ResNet-101 and ResNet-152 to SNNs. We demonstrate the effectiveness of our method on CIFAR-10 and ImageNet, achieving competitive performance. The code will be made available as open-source.

Noise Adaptor: Enhancing Low-Latency Spiking Neural Networks through Noise-Injected Low-Bit ANN Conversion

TL;DR

Noise Adaptor is presented, a novel method for constructing competitive low-latency spiking neural networks (SNNs) by converting noise-injected, low-bit artificial neural networks (ANNs) by converting noise-injected, low-bit artificial neural networks (ANNs).

Abstract

We present Noise Adaptor, a novel method for constructing competitive low-latency spiking neural networks (SNNs) by converting noise-injected, low-bit artificial neural networks (ANNs). This approach builds on existing ANN-to-SNN conversion techniques but offers several key improvements: (1) By injecting noise during quantized ANN training, Noise Adaptor better accounts for the dynamic differences between ANNs and SNNs, significantly enhancing SNN accuracy. (2) Unlike previous methods, Noise Adaptor does not require the application of run-time noise correction techniques in SNNs, thereby avoiding modifications to the spiking neuron model and control flow during inference. (3) Our method extends the capability of handling deeper architectures, achieving successful conversions of activation-quantized ResNet-101 and ResNet-152 to SNNs. We demonstrate the effectiveness of our method on CIFAR-10 and ImageNet, achieving competitive performance. The code will be made available as open-source.

Paper Structure

This paper contains 22 sections, 1 theorem, 25 equations, 3 figures, 3 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Fast SNNs.
  4. Noise in SNNs.
  5. Preliminary and Background
  6. Spiking Neuron and Synaptic Models.
  7. Quant-ANN-to-SNN Conversion.
  8. Noise Adaptor
  9. Formulation
  10. Training with Noise Injection
  11. Updations on the Conversion Process
  12. Connections to Existing Theoretical Work
  13. Experimental Results
  14. Experimental Setup
  15. Latency and Accuracy Improvements by Noise Adaptor
  16. ...and 7 more sections

Key Result

Theorem 1

Let $\hat{v}^l$ be the output of the function where $\epsilon$ is a uniformly distributed random variable over the interval $(-0.5, 0.5)$, i.e., $\epsilon \sim \mathcal{U}(-0.5, 0.5)$. The mean output $\mathbb{E}[\hat{v}^l]$ is given by:

Figures (3)

  • Figure 1: a. In the standard quant-ANN-to-SNN conversion process, ANNs are trained with activation quantization to model the discrete spikes characteristic of SNNs. b. Our method goes further by introducing controllable noise into the activation quantization, optimizing the training process for both quantization accuracy and noise resilience.
  • Figure 2: The response curve of spiking neurons for time steps $T=3$, $T=6$, $T=12$, and for time steps much larger than 12. As $T$ increases, the response smoothens and converges to a rectified-ReLU function.
  • Figure 3: SNN accuracy over time steps for ResNet-50 with quantization upper bounds $p$ set at 3, 7, and 15. Results show that a small $p$ initially improves accuracy more rapidly, but a higher $p$ may surpass it over time.

Theorems & Definitions (2)

  • Theorem 1
  • proof