FireFly-S: Exploiting Dual-Side Sparsity for Spiking Neural Networks Acceleration with Reconfigurable Spatial Architecture

TL;DR

FireFly-S addresses the challenge of efficiently accelerating Spiking Neural Networks on edge FPGA hardware by exploiting dual-side sparsity in weights and spikes. It integrates pruning via gradient rewiring with a hardware-aware LSQ variant, enabling $>85\%$ weight sparsity at $4$-bit precision while maintaining accuracy, and couples this with a Bitmap-based dual-side sparsity detector in a fully on-chip, inter-layer pipelined spatial architecture. The hardware design uses a Dataflow Orchestrator to align data movement and an optimized Sparse Detector to skip zero computations, achieving high FPS/W across MNIST, DVS-Gesture, and CIFAR-10 without off-chip memory access. This co-design yields strong energy efficiency gains and scalability, demonstrating practical deployment of compressed SNNs on FPGA edge devices with competitive accuracy and throughput.

Abstract

Spiking Neural Networks (SNNs), with their brain-inspired structure using discrete spikes instead of continuous activations, are gaining attention for their potential of efficient processing on neuromorphic chips. While current SNN hardware accelerators often prioritize temporal spike sparsity, exploiting sparse synaptic weights offers significant untapped potential for even greater efficiency. To address this, we propose FireFly-S, a Sparse extension of the FireFly series. This co-optimized software-hardware design focusing on leveraging dual-side sparsity for acceleration. On the software side, we propose a novel algorithmic optimization framework that combines gradient rewiring for pruning and modified Learned Step Size Quantization (LSQ) tailored for SNNs, which achieves remarkable weight sparsity exceeding 85\% and enables efficient 4-bit quantization with negligible accuracy loss. On the hardware side, we present an efficient dual-side sparsity detector employing a Bitmap-based sparse decoding logic to pinpoint the positions of non-zero weights and input spikes. The logic allows for the direct bypassing of redundant computations, thereby enhancing computational efficiency. Different from the overlay architecture adopted by previous FireFly series, we adopt a spatial architecture with inter-layer pipelining that can fully exploit the nature of Field-Programmable Gate Arrays (FPGAs). A spatial-temporal dataflow is also proposed to support such inter-layer pipelining and avoid long-term temporal dependencies. In experiments conducted on the MNIST, DVS-Gesture and CIFAR-10 datasets, the FireFly-S model achieves 85-95\% sparsity with 4-bit quantization and the hardware accelerator effectively leverages the dual-side sparsity, delivering outstanding performance metrics of 10,047 FPS/W on MNIST, 3,683 FPS/W on DVS-Gesture, and 2,327 FPS/W on CIFAR-10.

Figures (8)

• Figure 1: Overview of the FireFly-S system architecture, combining the software optimization framework and hardware accelerator.
• Figure 2: Compression process in FireFly-S. A) Pruning process. B) Quantization process, including both inference and training stages.
• Figure 3: Comparison of the dataflow in FireFly-S to its previous version FireFly and FireFly v2. A) FireFly's dataflow design. B) FireFly v2's dataflow design. C) FireFly-S's dataflow design.
• Figure 4: Throughput-to-storage ratio versus sparsity for COO, CSR, and Bitmap sparse representations and dense representation. Data points are plotted at 0.01 sparsity intervals.
• Figure 5: Inter-layer Pipelined Architecture of FireFly-S. All cores are mapped onto a single FPGA chip, with each core acting as a pipeline stage responsible for the inference of one layer. These stages integrate modules for Padding, Dataflow Orchestration, Dual-side Sparsity Detection, and Maxpooling.