Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

FastSpiker: Enabling Fast Training for Spiking Neural Networks on Event-based Data through Learning Rate Enhancements for Autonomous Embedded Systems

Iqra Bano, Rachmad Vidya Wicaksana Putra, Alberto Marchisio, Muhammad Shafique

TL;DR

FastSpiker tackles the long training times of spiking neural networks (SNNs) on event-based data for autonomous embedded systems. It proposes a three-step methodology to identify and tune learning rate policies, focusing on exponential decay and warm restarts, evaluated on the NCARS dataset with STBP training. The approach achieves up to 10.5x faster training and up to 88.39% carbon reduction while maintaining or improving accuracy compared with the state-of-the-art. This work enables greener, faster deployment of embodied neuromorphic intelligence in resource-constrained environments.

Abstract

Autonomous embedded systems (e.g., robots) typically necessitate intelligent computation with low power/energy processing for completing their tasks. Such requirements can be fulfilled by embodied neuromorphic intelligence with spiking neural networks (SNNs) because of their high learning quality (e.g., accuracy) and sparse computation. Here, the employment of event-based data is preferred to ensure seamless connectivity between input and processing parts. However, state-of-the-art SNNs still face a long training time to achieve high accuracy, thereby incurring high energy consumption and producing a high rate of carbon emission. Toward this, we propose FastSpiker, a novel methodology that enables fast SNN training on event-based data through learning rate enhancements targeting autonomous embedded systems. In FastSpiker, we first investigate the impact of different learning rate policies and their values, then select the ones that quickly offer high accuracy. Afterward, we explore different settings for the selected learning rate policies to find the appropriate policies through a statistical-based decision. Experimental results show that our FastSpiker offers up to 10.5x faster training time and up to 88.39% lower carbon emission to achieve higher or comparable accuracy to the state-of-the-art on the event-based automotive dataset (i.e., NCARS). In this manner, our FastSpiker methodology paves the way for green and sustainable computing in realizing embodied neuromorphic intelligence for autonomous embedded systems.

FastSpiker: Enabling Fast Training for Spiking Neural Networks on Event-based Data through Learning Rate Enhancements for Autonomous Embedded Systems

TL;DR

FastSpiker tackles the long training times of spiking neural networks (SNNs) on event-based data for autonomous embedded systems. It proposes a three-step methodology to identify and tune learning rate policies, focusing on exponential decay and warm restarts, evaluated on the NCARS dataset with STBP training. The approach achieves up to 10.5x faster training and up to 88.39% carbon reduction while maintaining or improving accuracy compared with the state-of-the-art. This work enables greener, faster deployment of embodied neuromorphic intelligence in resource-constrained environments.

Abstract

Autonomous embedded systems (e.g., robots) typically necessitate intelligent computation with low power/energy processing for completing their tasks. Such requirements can be fulfilled by embodied neuromorphic intelligence with spiking neural networks (SNNs) because of their high learning quality (e.g., accuracy) and sparse computation. Here, the employment of event-based data is preferred to ensure seamless connectivity between input and processing parts. However, state-of-the-art SNNs still face a long training time to achieve high accuracy, thereby incurring high energy consumption and producing a high rate of carbon emission. Toward this, we propose FastSpiker, a novel methodology that enables fast SNN training on event-based data through learning rate enhancements targeting autonomous embedded systems. In FastSpiker, we first investigate the impact of different learning rate policies and their values, then select the ones that quickly offer high accuracy. Afterward, we explore different settings for the selected learning rate policies to find the appropriate policies through a statistical-based decision. Experimental results show that our FastSpiker offers up to 10.5x faster training time and up to 88.39% lower carbon emission to achieve higher or comparable accuracy to the state-of-the-art on the event-based automotive dataset (i.e., NCARS). In this manner, our FastSpiker methodology paves the way for green and sustainable computing in realizing embodied neuromorphic intelligence for autonomous embedded systems.
Paper Structure (24 sections, 3 equations, 11 figures, 3 tables, 6 algorithms)

This paper contains 24 sections, 3 equations, 11 figures, 3 tables, 6 algorithms.

Table of Contents

  1. Introduction
  2. State-of-the-art and Their Limitations
  3. Motivational Case Study
  4. Our Novel Contributions
  5. Background
  6. Spiking Neural Networks (SNNs)
  7. Learning Rate Policies
  8. Decreasing Step
  9. Exponential Decay
  10. One-Cycle
  11. Cyclical
  12. Decreasing Cyclical
  13. Warm Restarts
  14. Event-based Data
  15. Carbon Emission of NN Training
  16. ...and 9 more sections

Figures (11)

  • Figure 1: Accuracy profile of the state-of-the-art SNN model considering the NCARS dataset; adapted from Ref_Viale_CarSNN_IJCNN21. Its accuracy saturates within 1% of the standard deviation after reaching at least around 80 training epochs.
  • Figure 2: Learning rate policies considered in the case study: (a) decreasing step, and (b) 4-peak warm restarts. Note, if a training phase is defined as 200 training epochs, then the 0.5 training phase means 100 training epochs. (c) Accuracy profiles from the decreasing step and warm restarts policies.
  • Figure 3: Our novel contributions in this paper, highlighted in purple.
  • Figure 4: Different types of LR policies: (a) decreasing step, (b) exponential decay, (c) one-cycle, (d) cyclical, (e) decreasing cyclical, and (f) warm restarts.
  • Figure 5: Illustration of the NCARS dataset Ref_Sironi_HATS_CVPR18Ref_Viale_CarSNN_IJCNN21.
  • ...and 6 more figures