A Survey on Structure-Preserving Graph Transformers

TL;DR

This survey addresses the challenge that vanilla graph transformers may overlook graph structure due to permutation-equivariant self-attention. It offers a fourfold taxonomy of structure-preserving strategies—node feature modulation, context node sampling, graph rewriting, and transformer architecture improvements—and surveys concrete techniques such as Laplacian-based positional encodings, random-walk encodings, subgraph sampling, coarsening, and GNN-augmented transformers. The authors map methods to these categories, discuss patterns, and analyze limitations, including the balance between structural identification and structural similarity, the need for geometric/equivariant properties, and scalability concerns with quadratic complexity $O(N^2)$. The work guides the design of more expressive, scalable, and geometry-aware graph transformers for chemoinformatics, bioinformatics, and related graph domains.

Abstract

The transformer architecture has shown remarkable success in various domains, such as natural language processing and computer vision. When it comes to graph learning, transformers are required not only to capture the interactions between pairs of nodes but also to preserve graph structures connoting the underlying relations and proximity between them, showing the expressive power to capture different graph structures. Accordingly, various structure-preserving graph transformers have been proposed and widely used for various tasks, such as graph-level tasks in bioinformatics and chemoinformatics. However, strategies related to graph structure preservation have not been well organized and systematized in the literature. In this paper, we provide a comprehensive overview of structure-preserving graph transformers and generalize these methods from the perspective of their design objective. First, we divide strategies into four main groups: node feature modulation, context node sampling, graph rewriting, and transformer architecture improvements. We then further divide the strategies according to the coverage and goals of graph structure preservation. Furthermore, we also discuss challenges and future directions for graph transformer models to preserve the graph structure and understand the nature of graphs.

Figures (6)

• Figure 1: Same molecular formulas, but different structures.
• Figure 2: An overview of structure preservation strategies of graph transformers.
• Figure 3: Two main approaches of node feature modulation.
• Figure 4: Two main approaches of context node sampling.
• Figure 5: Three main approaches of graph rewriting.