Bishop: Sparsified Bundling Spiking Transformers on Heterogeneous Cores with Error-Constrained Pruning

TL;DR

Bishop tackles the challenge of accelerating spiking transformers by introducing Token-Time Bundles (TTBs) to pack spiking activity across tokens and time, enabling structured data reuse on a heterogeneous core array. The framework combines a TT-Bundle Dense Core, TT-Bundle Sparse Core, and a TT-Bundle Attention Core, augmented by Bundle-Sparsity Aware Training (BSA) and Error-Constrained TT Bundle Pruning (ECP) to maximize sparsity and prune attention workloads within a controlled error bound. Empirical results show substantial improvements, including average speedups of up to 5.91x over an edge GPU and 299x over PTB, and energy reductions up to 6.11x over PTB, while maintaining or improving accuracy across CIFAR-10/100, ImageNet-100, DVS-Gesture-128, and Speech Command tasks. These findings demonstrate a practical, hardware-software co-design path for efficient spiking transformer inference on heterogeneous hardware platforms, enabling scalable deployment of energy-efficient SNN-based transformers.

Abstract

We present Bishop, the first dedicated hardware accelerator architecture and HW/SW co-design framework for spiking transformers that optimally represents, manages, and processes spike-based workloads while exploring spatiotemporal sparsity and data reuse. Specifically, we introduce the concept of Token-Time Bundle (TTB), a container that bundles spiking data of a set of tokens over multiple time points. Our heterogeneous accelerator architecture Bishop concurrently processes workload packed in TTBs and explores intra- and inter-bundle multiple-bit weight reuse to significantly reduce memory access. Bishop utilizes a stratifier, a dense core array, and a sparse core array to process MLP blocks and projection layers. The stratifier routes high-density spiking activation workload to the dense core and low-density counterpart to the sparse core, ensuring optimized processing tailored to the given spatiotemporal sparsity level. To further reduce data access and computation, we introduce a novel Bundle Sparsity-Aware (BSA) training pipeline that enhances not only the overall but also structured TTB-level firing sparsity. Moreover, the processing efficiency of self-attention layers is boosted by the proposed Error-Constrained TTB Pruning (ECP), which trims activities in spiking queries, keys, and values both before and after the computation of spiking attention maps with a well-defined error bound. Finally, we design a reconfigurable TTB spiking attention core to efficiently compute spiking attention maps by executing highly simplified "AND" and "Accumulate" operations. On average, Bishop achieves a 5.91x speedup and 6.11x improvement in energy efficiency over previous SNN accelerators, while delivering higher accuracy across multiple datasets.

Figures (17)

• Figure 1: Bishop: the first accelerator architecture and SW/HW co-design framework dedicated to spiking transformers.
• Figure 2: Spiking transformer model architecture with multi-head spiking self-attention yao2024spikespikformer.
• Figure 3: The FLOPs breakdown of a spiking transformer with different token and feature sizes trained on ImageNet.
• Figure 4: (a) The conventional approach lacks parallel processing in time and space, processing each token at each timepoint in a time-serial manner, causing irregularly repeated weight accesses. (b) The spiking Token-Time (TT)-Bundle allows for multi-bit weight data reuse across multiple tokens and time points. (c) Weight reuse in both an intra- and inter-bundle manner. Multiple TT bundles of neurons corresponding to different output features are mapped onto different PE columns. Multiple TT bundles across different input features marked as different color depths are mapped onto different PE rows.
• Figure 5: Active bundle distribution of spiking queries (Q) across each input feature in the $4^{th}$ encoder block of a spiking ViT (Model 1) trained on CIFAR10: (a) without BSA, and (b) with BSA.