Matterhorn: Efficient Analog Sparse Spiking Transformer Architecture with Masked Time-To-First-Spike Encoding

TL;DR

This work tackles the energy bottleneck of spiking neural networks for large-scale language tasks by reframing encoding and computation to minimize data movement. It introduces masked time-to-first-spike (M-TTFS) encoding, which remaps the silent state to the most frequent activation (via $I_{\max}$) and adds a tunable dead zone $k$ to further increase sparsity, paired with a Memristive Synapse Unit (MSU) that performs in-memory analog integration. The combination yields a spiking transformer that not only achieves state-of-the-art GLUE performance among SNNs but also delivers substantial energy savings (notably a $\approx$ $2.31\times$ improvement over prior SOTA) through both algorithmic sparsity and CIM-based computation. The results demonstrate practical viability of analog, sparse SNNs for energy-efficient, large-scale NLP on chip-capable hardware, with a compact area footprint and strong robustness characteristics to hardware non-idealities.

Abstract

Spiking neural networks (SNNs) have emerged as a promising candidate for energy-efficient LLM inference. However, current energy evaluations for SNNs primarily focus on counting accumulate operations, and fail to account for real-world hardware costs such as data movement, which can consume nearly 80% of the total energy. In this paper, we propose Matterhorn, a spiking transformer that integrates a novel masked time-to-first-spike (M-TTFS) encoding method to reduce spike movement and a memristive synapse unit (MSU) to eliminate weight access overhead. M-TTFS employs a masking strategy that reassigns the zero-energy silent state (a spike train of all 0s) to the most frequent membrane potential rather than the lowest. This aligns the coding scheme with the data distribution, minimizing spike movement energy without information loss. We further propose a `dead zone' strategy that maximizes sparsity by mapping all values within a given range to the silent state. At the hardware level, the MSU utilizes compute-in-memory (CIM) technology to perform analog integration directly within memory, effectively removing weight access costs. On the GLUE benchmark, Matterhorn establishes a new state-of-the-art, surpassing existing SNNs by 1.42% in average accuracy while delivering a 2.31 times improvement in energy efficiency.

Figures (14)

• Figure 1: Proportional Energy Breakdown of SOTA spiking transformers. Otters (standard TTFS encoding version is chose), Sorbet (rate encoding), SpikingBERT (rate encoding), and SpikingLM (rate encoding) are evaluated with time steps $T$ set to 15, 16, 16, and 4, and average spike rates of 5.14%, 13%, 25%, and 33%, respectively, as reported in bal2024spikingbertxingspikelmtangsorbetyan2025otters. SNN models are assumed to be deployed on a spatial dataflow architecture as described in Section \ref{['sec: energy_cal']}.
• Figure 2: M-TTFS with dead zone.
• Figure 3: M-TTFS spiking transformer structure.
• Figure 4: Memristive Synapse Unit (MSU) architecture.
• Figure 5: Comparison of spike time distributions on the SST-2 dataset. We compare the traditional TTFS baseline (a) with our proposed M-TTFS method under different dead-zone parameters.