Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Twin Network Augmentation: A Novel Training Strategy for Improved Spiking Neural Networks and Efficient Weight Quantization

Lucas Deckers, Benjamin Vandersmissen, Ing Jyh Tsang, Werner Van Leekwijck, Steven Latré

TL;DR

Twin Network Augmentation (TNA), a novel training framework aimed at improving the performance of SNNs while also facilitating an enhanced compression through low-precision quantization of weights, is presented.

Abstract

The proliferation of Artificial Neural Networks (ANNs) has led to increased energy consumption, raising concerns about their sustainability. Spiking Neural Networks (SNNs), which are inspired by biological neural systems and operate using sparse, event-driven spikes to communicate information between neurons, offer a potential solution due to their lower energy requirements. An alternative technique for reducing a neural network's footprint is quantization, which compresses weight representations to decrease memory usage and energy consumption. In this study, we present Twin Network Augmentation (TNA), a novel training framework aimed at improving the performance of SNNs while also facilitating an enhanced compression through low-precision quantization of weights. TNA involves co-training an SNN with a twin network, optimizing both networks to minimize their cross-entropy losses and the mean squared error between their output logits. We demonstrate that TNA significantly enhances classification performance across various vision datasets and in addition is particularly effective when applied when reducing SNNs to ternary weight precision. Notably, during inference , only the ternary SNN is retained, significantly reducing the network in number of neurons, connectivity and weight size representation. Our results show that TNA outperforms traditional knowledge distillation methods and achieves state-of-the-art performance for the evaluated network architecture on benchmark datasets, including CIFAR-10, CIFAR-100, and CIFAR-10-DVS. This paper underscores the effectiveness of TNA in bridging the performance gap between SNNs and ANNs and suggests further exploration into the application of TNA in different network architectures and datasets.

Twin Network Augmentation: A Novel Training Strategy for Improved Spiking Neural Networks and Efficient Weight Quantization

TL;DR

Twin Network Augmentation (TNA), a novel training framework aimed at improving the performance of SNNs while also facilitating an enhanced compression through low-precision quantization of weights, is presented.

Abstract

The proliferation of Artificial Neural Networks (ANNs) has led to increased energy consumption, raising concerns about their sustainability. Spiking Neural Networks (SNNs), which are inspired by biological neural systems and operate using sparse, event-driven spikes to communicate information between neurons, offer a potential solution due to their lower energy requirements. An alternative technique for reducing a neural network's footprint is quantization, which compresses weight representations to decrease memory usage and energy consumption. In this study, we present Twin Network Augmentation (TNA), a novel training framework aimed at improving the performance of SNNs while also facilitating an enhanced compression through low-precision quantization of weights. TNA involves co-training an SNN with a twin network, optimizing both networks to minimize their cross-entropy losses and the mean squared error between their output logits. We demonstrate that TNA significantly enhances classification performance across various vision datasets and in addition is particularly effective when applied when reducing SNNs to ternary weight precision. Notably, during inference , only the ternary SNN is retained, significantly reducing the network in number of neurons, connectivity and weight size representation. Our results show that TNA outperforms traditional knowledge distillation methods and achieves state-of-the-art performance for the evaluated network architecture on benchmark datasets, including CIFAR-10, CIFAR-100, and CIFAR-10-DVS. This paper underscores the effectiveness of TNA in bridging the performance gap between SNNs and ANNs and suggests further exploration into the application of TNA in different network architectures and datasets.
Paper Structure (19 sections, 6 equations, 3 figures, 3 tables)

This paper contains 19 sections, 6 equations, 3 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related work
  3. Preliminary
  4. Neuron models
  5. Training SNN
  6. Methodology
  7. Twin network augmentation in spiking neural networks
  8. Ternary spiking neural networks
  9. Experiments
  10. Setup
  11. Datasets
  12. Training details
  13. Results
  14. Network augmentation effectively regularizes SNN
  15. The role of the balancing parameter alpha
  16. ...and 4 more sections

Figures (3)

  • Figure 1: The overall workflow of the proposed novel algorithm is as follows: During the training phase, a twin network is instantiated and co-trained from scratch alongside the base SNN. The training loss comprises three components: the two individual cross-entropy losses of the two networks and the logit matching loss. In the inference phase, only the original base network is employed to generate predictions.
  • Figure 2: The significance of selecting an appropriate balancing parameter $\alpha$. We show the loss on the CIFAR-100 training (a) and validation set (d) as well as the classification accuracy on the training set (b) and validation set (e). In (c) we show the magnitude of the matching loss.
  • Figure 3: (a) Hyperparameter $\alpha$ is used to balance between the CEloss and the KD logit matching loss. We found an $\alpha$ of $1.e^{-5}$ performs best in this particular scenario. (b) Comparison of the KD-based methods with our baseline SNN and the new twin network augmentation SNN on the CIFAR-100 dataset.