Transformer for Graphs: An Overview from Architecture Perspective
Erxue Min, Runfa Chen, Yatao Bian, Tingyang Xu, Kangfei Zhao, Wenbing Huang, Peilin Zhao, Junzhou Huang, Sophia Ananiadou, Yu Rong
TL;DR
Graph data are non-Euclidean, and standard Transformers lack graph-aware inductive biases. The paper provides a systematic taxonomy of Graph Transformer variants—GNNs as Auxiliary Modules (GA), Improved Positional Embeddings (PE), and Improved Attention (AT)—and conducts ablations across six benchmarks to quantify gains. Key findings show that graph-aware components generally improve performance, with GA and AT delivering larger gains than PE, and graph-level tasks benefiting more than node-level tasks. The work offers practical guidance on which integration pattern to favor per task type and outlines directions for scalable, heterogeneous, and large-scale Graph-Transformer research.
Abstract
Recently, Transformer model, which has achieved great success in many artificial intelligence fields, has demonstrated its great potential in modeling graph-structured data. Till now, a great variety of Transformers has been proposed to adapt to the graph-structured data. However, a comprehensive literature review and systematical evaluation of these Transformer variants for graphs are still unavailable. It's imperative to sort out the existing Transformer models for graphs and systematically investigate their effectiveness on various graph tasks. In this survey, we provide a comprehensive review of various Graph Transformer models from the architectural design perspective. We first disassemble the existing models and conclude three typical ways to incorporate the graph information into the vanilla Transformer: 1) GNNs as Auxiliary Modules, 2) Improved Positional Embedding from Graphs, and 3) Improved Attention Matrix from Graphs. Furthermore, we implement the representative components in three groups and conduct a comprehensive comparison on various kinds of famous graph data benchmarks to investigate the real performance gain of each component. Our experiments confirm the benefits of current graph-specific modules on Transformer and reveal their advantages on different kinds of graph tasks.