Spiking Transformer:Introducing Accurate Addition-Only Spiking Self-Attention for Transformer

TL;DR

This work tackles the high energy demand of Transformer self-attention by introducing Accurate Addition-Only Spiking Self-Attention (A$^2$OS$^2$A), a hybrid spiking mechanism that uses binary Q, full-precision K, and ternary V to enable addition-only computations without softmax or scaling. The authors provide a theoretical analysis of information loss in purely binary spiking attention and demonstrate that enriching Q/K/V preserves representational capacity while maintaining energy efficiency. Empirical results on CIFAR and ImageNet-1K show improvements over prior SNN-based Transformers, with a reported top-1 accuracy of up to 78.66% on ImageNet-1K and strong gains on CIFAR datasets. Overall, A$^2$OS$^2$A advances energy-efficient vision transformers by coupling neuromorphic-inspired spiking dynamics with a more expressive self-attention mechanism, enabling practical deployment on real-world tasks.

Abstract

Transformers have demonstrated outstanding performance across a wide range of tasks, owing to their self-attention mechanism, but they are highly energy-consuming. Spiking Neural Networks have emerged as a promising energy-efficient alternative to traditional Artificial Neural Networks, leveraging event-driven computation and binary spikes for information transfer. The combination of Transformers' capabilities with the energy efficiency of SNNs offers a compelling opportunity. This paper addresses the challenge of adapting the self-attention mechanism of Transformers to the spiking paradigm by introducing a novel approach: Accurate Addition-Only Spiking Self-Attention (A$^2$OS$^2$A). Unlike existing methods that rely solely on binary spiking neurons for all components of the self-attention mechanism, our approach integrates binary, ReLU, and ternary spiking neurons. This hybrid strategy significantly improves accuracy while preserving non-multiplicative computations. Moreover, our method eliminates the need for softmax and scaling operations. Extensive experiments show that the A$^2$OS$^2$A-based Spiking Transformer outperforms existing SNN-based Transformers on several datasets, even achieving an accuracy of 78.66\% on ImageNet-1K. Our work represents a significant advancement in SNN-based Transformer models, offering a more accurate and efficient solution for real-world applications.

Figures (2)

• Figure 1: The difference between our Spiking Self-Attention and the vanilla Spiking Self-Attention. Our Spiking Self-Attention differs significantly from the vanilla Spiking Self-Attention. In the vanilla version, only binary spikes are employed, which can result in considerable information loss. In contrast, our approach integrates a combination of binary spiking neurons, ReLU, and ternary spiking neurons. This hybrid structure effectively reduces information loss, while preserving the advantages of addition-only processing. Moreover, our method eliminates the need for both scaling and the softmax function, streamlining the computation.
• Figure 2: The overview of Spiking Transformer.

Theorems & Definitions (1)

• Proposition 1