Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Linearized Bregman Iterations for Sparse Spiking Neural Networks

Daniel Windhager, Bernhard A. Moser, Michael Lunglmayr

Abstract

Spiking Neural Networks (SNNs) offer an energy efficient alternative to conventional Artificial Neural Networks (ANNs) but typically still require a large number of parameters. This work introduces Linearized Bregman Iterations (LBI) as an optimizer for training SNNs, enforcing sparsity through iterative minimization of the Bregman distance and proximal soft thresholding updates. To improve convergence and generalization, we employ the AdaBreg optimizer, a momentum and bias corrected Bregman variant of Adam. Experiments on three established neuromorphic benchmarks, i.e. the Spiking Heidelberg Digits (SHD), the Spiking Speech Commands (SSC), and the Permuted Sequential MNIST (PSMNIST) datasets, show that LBI based optimization reduces the number of active parameters by about 50% while maintaining accuracy comparable to models trained with the Adam optimizer, demonstrating the potential of convex sparsity inducing methods for efficient neuromorphic learning.

Linearized Bregman Iterations for Sparse Spiking Neural Networks

Abstract

Spiking Neural Networks (SNNs) offer an energy efficient alternative to conventional Artificial Neural Networks (ANNs) but typically still require a large number of parameters. This work introduces Linearized Bregman Iterations (LBI) as an optimizer for training SNNs, enforcing sparsity through iterative minimization of the Bregman distance and proximal soft thresholding updates. To improve convergence and generalization, we employ the AdaBreg optimizer, a momentum and bias corrected Bregman variant of Adam. Experiments on three established neuromorphic benchmarks, i.e. the Spiking Heidelberg Digits (SHD), the Spiking Speech Commands (SSC), and the Permuted Sequential MNIST (PSMNIST) datasets, show that LBI based optimization reduces the number of active parameters by about 50% while maintaining accuracy comparable to models trained with the Adam optimizer, demonstrating the potential of convex sparsity inducing methods for efficient neuromorphic learning.
Paper Structure (9 sections, 4 equations, 5 figures, 2 tables)

This paper contains 9 sections, 4 equations, 5 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Linearized Bregman Iterations
  3. AdaBreg Optimization
  4. Results
  5. Setup and configurations
  6. Performance with learning rate schedulers
  7. Performance without learning rate schedulers
  8. Performance compared to training with Adam
  9. Conclusion

Figures (5)

  • Figure 1: Loss curves for the training on SHD, SSC and PSMNIST dataset with different values for $\lambda$. All curves were averaged over three separate training runs with different seeds.
  • Figure 2: Number of non-zero values in networks for SHD, SSC and PSMNIST datasets during the training process plotted as a function of current epoch for different $\lambda$ values. Results represent the mean across three independent training runs.
  • Figure 3: Peak validation accuracy across SHD, SSC, and PSMNIST datasets as a function of regularization parameter $\lambda$, averaged over multiple training runs. The optimal $\lambda$ is slightly higher for SHD and SSC, while a lower value of $\lambda$ achieves the best results for PSMNIST.
  • Figure 4: Loss curves for the training on SHD, SSC and PSMNIST dataset with different values for $\lambda$. All curves represent means over three independent training runs with different random seeds.
  • Figure 5: Peak validation accuracy without learning rate scheduling across SHD, SSC, and PSMNIST datasets versus regularization parameter $\lambda$, averaged over multiple training runs.