Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Graph Transformers: A Survey

Ahsan Shehzad, Feng Xia, Shagufta Abid, Ciyuan Peng, Shuo Yu, Dongyu Zhang, Karin Verspoor

TL;DR

Graph Transformers surveys how self-attention augmented with graph inductive biases can model complex graph-structured data. It provides a four-way taxonomy (shallow, deep, scalable, pre-trained) and a design guide that links architectural choices to graph properties and tasks. The paper catalogs applications across node-, edge-, and graph-level tasks, including protein modeling, KG completion, and molecular generation, and highlights cross-domain uses in NLP and CV. It also outlines key open issues—scalability, generalization, interpretability, and learning on dynamic graphs—along with practical directions for future research and deployment.

Abstract

Graph transformers are a recent advancement in machine learning, offering a new class of neural network models for graph-structured data. The synergy between transformers and graph learning demonstrates strong performance and versatility across various graph-related tasks. This survey provides an in-depth review of recent progress and challenges in graph transformer research. We begin with foundational concepts of graphs and transformers. We then explore design perspectives of graph transformers, focusing on how they integrate graph inductive biases and graph attention mechanisms into the transformer architecture. Furthermore, we propose a taxonomy classifying graph transformers based on depth, scalability, and pre-training strategies, summarizing key principles for effective development of graph transformer models. Beyond technical analysis, we discuss the applications of graph transformer models for node-level, edge-level, and graph-level tasks, exploring their potential in other application scenarios as well. Finally, we identify remaining challenges in the field, such as scalability and efficiency, generalization and robustness, interpretability and explainability, dynamic and complex graphs, as well as data quality and diversity, charting future directions for graph transformer research.

Graph Transformers: A Survey

TL;DR

Graph Transformers surveys how self-attention augmented with graph inductive biases can model complex graph-structured data. It provides a four-way taxonomy (shallow, deep, scalable, pre-trained) and a design guide that links architectural choices to graph properties and tasks. The paper catalogs applications across node-, edge-, and graph-level tasks, including protein modeling, KG completion, and molecular generation, and highlights cross-domain uses in NLP and CV. It also outlines key open issues—scalability, generalization, interpretability, and learning on dynamic graphs—along with practical directions for future research and deployment.

Abstract

Graph transformers are a recent advancement in machine learning, offering a new class of neural network models for graph-structured data. The synergy between transformers and graph learning demonstrates strong performance and versatility across various graph-related tasks. This survey provides an in-depth review of recent progress and challenges in graph transformer research. We begin with foundational concepts of graphs and transformers. We then explore design perspectives of graph transformers, focusing on how they integrate graph inductive biases and graph attention mechanisms into the transformer architecture. Furthermore, we propose a taxonomy classifying graph transformers based on depth, scalability, and pre-training strategies, summarizing key principles for effective development of graph transformer models. Beyond technical analysis, we discuss the applications of graph transformer models for node-level, edge-level, and graph-level tasks, exploring their potential in other application scenarios as well. Finally, we identify remaining challenges in the field, such as scalability and efficiency, generalization and robustness, interpretability and explainability, dynamic and complex graphs, as well as data quality and diversity, charting future directions for graph transformer research.
Paper Structure (43 sections, 14 equations, 3 figures, 5 tables)

This paper contains 43 sections, 14 equations, 3 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Notations and Preliminaries
  3. Graphs and Graph Neural Networks
  4. Self-attention and Transformers
  5. Design Perspectives of Graph Transformers
  6. Graph Inductive Bias
  7. Node Positional Bias
  8. Edge Structural Bias
  9. Message-passing Bias
  10. Attention Bias
  11. Graph Attention Mechanisms
  12. Global Attention Mechanisms
  13. Local Attention Mechanisms
  14. Taxonomy of Graph Transformers
  15. Shallow Graph Transformers
  16. ...and 28 more sections

Figures (3)

  • Figure 1: Organization of this paper
  • Figure 2: An illustration of transformer architecture vaswaniAttentionAllYou2017a.
  • Figure 3: Incorporation of message-passing bias in transformer model.