Parallel Hyperparameter Optimization Of Spiking Neural Network
Thomas Firmin, Pierre Boulet, El-Ghazali Talbi
TL;DR
The paper tackles the challenging problem of hyperparameter optimization for Spiking Neural Networks by introducing silent networks and a spike-based early stopping mechanism, coupled with black-box constraints, to safely explore high-dimensional search spaces. A scalable Scalable Constrained Bayesian Optimization (SCBO) framework with asynchronous Thompson sampling and trust-region ideas enables efficient optimization across STDP and surrogate-gradient training on large multi-GPU systems. Large-scale experiments on MNIST and DVS Gesture demonstrate that permitting silent networks can accelerate search without sacrificing final performance, and that the approach generalizes to multiple training paradigms and decoders. The work suggests promising directions for multi-objective HPO, multi-fidelity strategies, and potential integration with Neural Architecture Search for SNNs, with practical implications for energy-efficient neuromorphic systems.
Abstract
Spiking Neural Networks (SNN). SNNs are based on a more biologically inspired approach than usual artificial neural networks. Such models are characterized by complex dynamics between neurons and spikes. These are very sensitive to the hyperparameters, making their optimization challenging. To tackle hyperparameter optimization of SNNs, we initially extended the signal loss issue of SNNs to what we call silent networks. These networks fail to emit enough spikes at their outputs due to mistuned hyperparameters or architecture. Generally, search spaces are heavily restrained, sometimes even discretized, to prevent the sampling of such networks. By defining an early stopping criterion detecting silent networks and by designing specific constraints, we were able to instantiate larger and more flexible search spaces. We applied a constrained Bayesian optimization technique, which was asynchronously parallelized, as the evaluation time of a SNN is highly stochastic. Large-scale experiments were carried-out on a multi-GPU Petascale architecture. By leveraging silent networks, results show an acceleration of the search, while maintaining good performances of both the optimization algorithm and the best solution obtained. We were able to apply our methodology to two popular training algorithms, known as spike timing dependent plasticity and surrogate gradient. Early detection allowed us to prevent worthless and costly computation, directing the search toward promising hyperparameter combinations. Our methodology could be applied to multi-objective problems, where the spiking activity is often minimized to reduce the energy consumption. In this scenario, it becomes essential to find the delicate frontier between low-spiking and silent networks. Finally, our approach may have implications for neural architecture search, particularly in defining suitable spiking architectures.