Graph Fourier Transformer with Structure-Frequency Information

TL;DR

This paper addresses the limitations of Graph Transformers in capturing both local structural details and global graph statistics by introducing Grafourierformer, which embeds Graph Fourier Transform-based structure and frequency cues into self-attention. It introduces two key components: a Laplacian eigenvalue mask that encodes structural relationships, and a node frequency energy filter that separates and reweights low- and high-frequency information to refine attention weights. Empirically, Grafourierformer surpasses 15 baselines across eight benchmarks, with notable gains in low-resource settings, and ablations confirm the critical roles of the frequency filter and spectral masks. The work demonstrates that joint structure-frequency inductive biases yield more robust, multi-scale graph representations and offers a practical route to improved performance in data-scarce regimes.

Abstract

Graph Transformers (GTs) have shown advantages in numerous graph structure tasks but their self-attention mechanism ignores the generalization bias of graphs, with existing methods mainly compensating for this bias from aspects like position encoding, attention bias and relative distance yet still having sub-optimal performance and being insufficient by only considering the structural perspective of generalization bias. To address this, this paper proposes Grafourierformer, which innovatively combines GT with inductive bias containing Frequency-Structure information by applying Graph Fourier Transform to the Attention Matrix: specifically, eigenvalues from the Graph Laplacian matrix are used to construct an Eigenvalue matrix mask (reflecting node positions and structural relationships with neighboring nodes to enable consideration of node range structural characteristics and focus on local graph details), and inverse Fourier transform is employed to extract node high-frequency and low-frequency features, calculate low-frequency and high-frequency energy, and construct a node frequency-energy matrix to filter the eigenvalue matrix mask, allowing attention heads to incorporate both graph structural information and node frequency information optimization, adaptively distinguish global trends from local details, and effectively suppress redundant information interference. Extensive experiments on various benchmarks show Grafourierformer consistently outperforms GNN and GT-based models in graph classification and node classification tasks, with ablation experiments further validating the effectiveness and necessity of the method. Codes are available at https://github.com/Arichibald/Grafourierformer.git

Figures (4)

• Figure 1: The structure of Grafourierformer is optimized by Structure-Frequency Mask on the attention matrix to achieve the fusion of graph structure information and node frequency information with attention.
• Figure 2: Figure Fourier filtering mechanism. The blue matrix denotes the Laplace Matrix Eigenvalue Mask, where the intensity of the blue color indicates the degree of the eigenvalue. The red matrix denotes the Filter Matrix, where the intensity of the red color indicates the size of the mask. After applying the mask (represented by the red cells with a gradient), the attentional values in the masked cells are significantly optimized. With this attention mask, the self-attention mechanism becomes more sensitive to the structure-frequency features of the graph.
• Figure 3: Laplace Matrix Eigenvalue Mask, Frequency Filter Matrix, and Structure-Frequency Mask visualizations on the NCI1, PROTEINS, MUTAG, and ogbg-molhiv datasets, respectively. (We choose randomly selected plots with the same ordinal number on each dataset as examples, with darker colors representing larger values)
• Figure 4: Kernel densities of high-frequency energies and low-frequency energies of nodes in the training sets of the NCI, PROTEINS, MUTAG, IMDB-BINARY, PATTERN, and CLUSTER datasets.