Phi: Leveraging Pattern-based Hierarchical Sparsity for High-Efficiency Spiking Neural Networks

TL;DR

This work addresses the inefficiency of prior SNN accelerators that exploit only bit sparsity by uncovering and leveraging structured activation patterns inherent in SNNs. It introduces Phi, a pattern-based hierarchical sparsity framework with two levels: Level 1 uses offline pre-defined activation patterns to enable Pattern-Wise Products (PWPs) and substantial runtime reduction, while Level 2 adds element-wise corrections to capture residual activations. The approach combines a k-means-based pattern calibration, pattern-aware fine-tuning (PAFT), and a dedicated Phi architecture with a Preprocessor, L1/L2 processors, and a Spiking Neuron Array to process two sparsity levels on the fly. Experimental results show Phi delivering up to 3.45× speedup and 4.93× energy efficiency over state-of-the-art SNN accelerators, along with detailed design-space and generalizability analyses that highlight the practicality and broad applicability of pattern-based hierarchical sparsity in SNNs.

Abstract

Spiking Neural Networks (SNNs) are gaining attention for their energy efficiency and biological plausibility, utilizing 0-1 activation sparsity through spike-driven computation. While existing SNN accelerators exploit this sparsity to skip zero computations, they often overlook the unique distribution patterns inherent in binary activations. In this work, we observe that particular patterns exist in spike activations, which we can utilize to reduce the substantial computation of SNN models. Based on these findings, we propose a novel \textbf{pattern-based hierarchical sparsity} framework, termed \textbf{\textit{Phi}}, to optimize computation. \textit{Phi} introduces a two-level sparsity hierarchy: Level 1 exhibits vector-wise sparsity by representing activations with pre-defined patterns, allowing for offline pre-computation with weights and significantly reducing most runtime computation. Level 2 features element-wise sparsity by complementing the Level 1 matrix, using a highly sparse matrix to further reduce computation while maintaining accuracy. We present an algorithm-hardware co-design approach. Algorithmically, we employ a k-means-based pattern selection method to identify representative patterns and introduce a pattern-aware fine-tuning technique to enhance Level 2 sparsity. Architecturally, we design \textbf{\textit{Phi}}, a dedicated hardware architecture that efficiently processes the two levels of \textit{Phi} sparsity on the fly. Extensive experiments demonstrate that \textit{Phi} achieves a $3.45\times$ speedup and a $4.93\times$ improvement in energy efficiency compared to state-of-the-art SNN accelerators, showcasing the effectiveness of our framework in optimizing SNN computation.

Figures (12)

• Figure 1: Activation visualization for (a) normal distribution, (b) ViT DNN model, and (c) Spikformer SNN model, and comparison between (d) bit sparsity and (e) ours Phi sparsity.
• Figure 2: Phi Sparsity.
• Figure 3: Phi architecture overview.
• Figure 4: Preprocessor.
• Figure 5: Level 2 Processor