SGHormer: An Energy-Saving Graph Transformer Driven by Spikes

TL;DR

SGHormer tackles the energy inefficiency of Graph Transformers by replacing dense attention with a spiking neural computation framework. It introduces a Spiking Rectify Block (SRB) to recover embedding information and a Spiking Graph Attention Head (SGSA) to produce sparse, binarized attention via rate-coded spikes, enabling largely addition-based operations and reduced memory. Across graph classification benchmarks, SGHormer achieves competitive accuracy with dramatic energy savings, reporting an average improvement of around 153x over full-precision GTs and a smaller model footprint suitable for edge devices. This neuromorphic-guided approach demonstrates the practical viability of energy-efficient graph transformers and motivates future development of graph-centric spiking operators.

Abstract

Graph Transformers (GTs) with powerful representation learning ability make a huge success in wide range of graph tasks. However, the costs behind outstanding performances of GTs are higher energy consumption and computational overhead. The complex structure and quadratic complexity during attention calculation in vanilla transformer seriously hinder its scalability on the large-scale graph data. Though existing methods have made strides in simplifying combinations among blocks or attention-learning paradigm to improve GTs' efficiency, a series of energy-saving solutions originated from biologically plausible structures are rarely taken into consideration when constructing GT framework. To this end, we propose a new spiking-based graph transformer (SGHormer). It turns full-precision embeddings into sparse and binarized spikes to reduce memory and computational costs. The spiking graph self-attention and spiking rectify blocks in SGHormer explicitly capture global structure information and recover the expressive power of spiking embeddings, respectively. In experiments, SGHormer achieves comparable performances to other full-precision GTs with extremely low computational energy consumption. The results show that SGHomer makes a remarkable progress in the field of low-energy GTs.

Figures (3)

• Figure 1: Visualization results of spiking attention matrices and the full-precision vanilla self-attention matrix. We construct the experiments on a selected graph from ZINC. For spiking attention matrices cross multiple time steps ($\textbf{bottom}$), parts of spiking outputs have similar attention patterns as the full-precision attention ($\textbf{top}$) calculated by a softmax function.
• Figure 2: The framework of SGHormer
• Figure 3: Performances of SGHormer with different membrane potential threshold and time step on PATTERN and ogbg-molhiv datasets.