FireFly-T: High-Throughput Sparsity Exploitation for Spiking Transformer Acceleration with Dual-Engine Overlay Architecture

TL;DR

FireFly-T tackles the bottlenecks of accelerating spiking transformers by introducing a dual-engine overlay that decouples activation sparsity from spiking attention. It combines a high-throughput sparse engine with multi-lane decoders and a bank-conflict-free load balancer with a resource-efficient binary engine that employs implicit dataflow manipulation and LUT6-optimized AND-PopCount to implement spiking attention, coordinated by a flexible orchestrator. The latency-hiding pipeline overlaps Q, K, V projections with binary attention computations, reducing attention time complexity from $O(3 T_s L d^2 + 2 T_s L^2 d)$ to $O(3 T_s L d^2)$. On edge-FPGAs, FireFly-T demonstrates notable gains in energy efficiency and DSP utilization over prior SNN and transformer accelerators, illustrating practical deployment potential for spiking transformers.

Abstract

Spiking transformers are emerging as a promising architecture that combines the energy efficiency of Spiking Neural Networks (SNNs) with the powerful attention mechanisms of transformers. However, existing hardware accelerators lack support for spiking attention, exhibit limited throughput in exploiting fine-grained sparsity, and struggle with scalable parallelism in sparse computation. To address these, we propose FireFly-T, a dual-engine overlay architecture that integrates a sparse engine for activation sparsity and a binary engine for spiking attention. In the sparse engine, we propose a highthroughput sparse decoder that exploits fine-grained sparsity by concurrently extracting multiple non-zero spikes. To complement this, we introduce a scalable load balancing mechanism with weight dispatch and out-of-order execution, eliminating bank conflicts to support scalable multidimensional parallelism. In the binary engine, we leverage the byte-level write capability of SRAMs to efficiently manipulate the 3D dataflows required for spiking attention with minimal resource overhead. We also optimize the core AND-PopCount operation in spiking attention through a LUT6-based implementation, improving timing closure and reducing LUT utilization on Xilinx FPGAs. As an overlay architecture, FireFly-T further incorporates an orchestrator that dynamically manipulates input dataflows with flexible adaptation for diverse network topologies, while ensuring efficient resource utilization and maintaining high throughput. Experimental results demonstrate that our accelerator achieves $1.39\times$ and $2.40\times$ higher energy efficiency, as well as $4.21\times$ and $7.10\times$ greater DSP efficiency, compared to FireFly v2 and the transformer-enabled SpikeTA, respectively. These results highlight its potential as an efficient hardware platform for spiking transformer.

Figures (13)

• Figure 1: Overview of major limitations in hardware acceleration for spiking transformers, alongside the proposed solutions and features of our FireFly-T.
• Figure 2: Overview of typical spiking transformer shen2024enhancingcao2025binary, which begins with a SPS module, followed by $L$ encoder blocks and a classification head. Each encoder block includes a SSA, an MLP, and associated residual connections.
• Figure 3: Block diagram of the FireFly-T system architecture on a Zynq UltraScale+ MPSoC (KV260), highlighting key features and components of FireFly-T IP, alongside the PS-PL interface for control and data transfer.
• Figure 4: Illustration of FireFly-T's parallelism and dataflow, featuring the orchestrator, sparse engine, and binary engine.
• Figure 5: Spatial-temporal diagram of the latency-hiding pipeline showing overlapped spiking attention execution in the sparse and binary engine.