Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Spiking Synaptic Penalty: Appropriate Penalty Term for Energy-Efficient Spiking Neural Networks

Kazuma Suetake, Takuya Ushimaru, Ryuji Saiin, Yoshihide Sawada

TL;DR

Spiking neural networks offer energy advantages that erode as spike rates rise. The paper introduces a spiking synaptic penalty that directly optimizes the energy consumption metric via $E_{SNN} = p\,E_{AC}\,\mathbb{E}_{x\in X}[\Omega_{syn}]$, enabling energy-aware training without architectural changes. Empirical results across CNN7, VGG11, and ResNet18 on Fashion-MNIST, CIFAR-10, and CIFAR-100 show that the penalty (especially with $p=1$) reduces energy usage while preserving accuracy, and it remains robust to the choice of surrogate gradient and to conversion-based baselines. This approach provides a practical route to eco-friendly SNNs suitable for neuromorphic hardware by aligning training objectives with actual energy costs.

Abstract

Spiking neural networks (SNNs) are energy-efficient neural networks because of their spiking nature. However, as the spike firing rate of SNNs increases, the energy consumption does as well, and thus, the advantage of SNNs diminishes. Here, we tackle this problem by introducing a novel penalty term for the spiking activity into the objective function in the training phase. Our method is designed so as to optimize the energy consumption metric directly without modifying the network architecture. Therefore, the proposed method can reduce the energy consumption more than other methods while maintaining the accuracy. We conducted experiments for image classification tasks, and the results indicate the effectiveness of the proposed method, which mitigates the dilemma of the energy--accuracy trade-off.

Spiking Synaptic Penalty: Appropriate Penalty Term for Energy-Efficient Spiking Neural Networks

TL;DR

Spiking neural networks offer energy advantages that erode as spike rates rise. The paper introduces a spiking synaptic penalty that directly optimizes the energy consumption metric via , enabling energy-aware training without architectural changes. Empirical results across CNN7, VGG11, and ResNet18 on Fashion-MNIST, CIFAR-10, and CIFAR-100 show that the penalty (especially with ) reduces energy usage while preserving accuracy, and it remains robust to the choice of surrogate gradient and to conversion-based baselines. This approach provides a practical route to eco-friendly SNNs suitable for neuromorphic hardware by aligning training objectives with actual energy costs.

Abstract

Spiking neural networks (SNNs) are energy-efficient neural networks because of their spiking nature. However, as the spike firing rate of SNNs increases, the energy consumption does as well, and thus, the advantage of SNNs diminishes. Here, we tackle this problem by introducing a novel penalty term for the spiking activity into the objective function in the training phase. Our method is designed so as to optimize the energy consumption metric directly without modifying the network architecture. Therefore, the proposed method can reduce the energy consumption more than other methods while maintaining the accuracy. We conducted experiments for image classification tasks, and the results indicate the effectiveness of the proposed method, which mitigates the dilemma of the energy--accuracy trade-off.
Paper Structure (30 sections, 1 theorem, 16 equations, 14 figures, 20 tables, 1 algorithm)

This paper contains 30 sections, 1 theorem, 16 equations, 14 figures, 20 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Principle and Idea
  3. Main Contributions
  4. Related Work
  5. Spike Sparsification in Direct SNN Training
  6. Spike Sparsification via Conversion from ANN
  7. Neuron Sparsification
  8. Method
  9. Neuron Model and Surrogate Gradient
  10. Metric for Energy Consumption
  11. Spiking Synaptic Penalty
  12. Differences from Other Penalty Terms
  13. Normalization of Penalty Terms
  14. Experiment
  15. Experimental Setup
  16. ...and 15 more sections

Key Result

Theorem 3.4

The expected value of the layer-wise and total spiking synaptic penalties are precisely proportional to the layer-wise and total energy consumption metrics of SNNs: for arbitrary $p\geq1$.

Figures (14)

  • Figure 1: Comparison among layer-wise penalty terms. The $x$-axis represents the layer number, and the $y$-axis represents the ratio of some layer-wise metric or penalty term to some total metric or penalty term. The network architecture is CNN7 (App. \ref{['app:detail-of-network-architecture']}), and the computation details are described in Sec. \ref{['sec:method']}. Our penalty term (blue) is precisely proportional to the ground truth (gray).
  • Figure 2: (A) VGG11
  • Figure 3: (B) ResNet18
  • Figure 5: (A) Eq. \ref{['eqn:penalty-synops']}
  • Figure 6: (B) Eq. \ref{['eqn:surrogate-triangle']}
  • ...and 9 more figures

Theorems & Definitions (5)

  • Definition 3.1
  • Definition 3.2
  • Definition 3.3
  • Theorem 3.4
  • Remark 3.5