Generalizable Insights for Graph Transformers in Theory and Practice

TL;DR

This work addresses the lack of generalizable insights for graph transformers by introducing the Generalized-Distance Transformer (GDT), a standard-attention GT whose expressivity is tied to the Generalized-Distance Weisfeiler--Leman (GD-WL) framework. The GDT unifies node- and edge-level tokenization, explicit edge embeddings, and both absolute and relative positional embeddings to encode graph structure, proving that it can simulate GD-WL and thus achieve powerful graph discrimination when equipped with appropriate PEs. Through large-scale experiments across real-world domains (molecular property prediction, image-based object detection, code summarization) and synthetic algorithmic tasks, the authors identify a practical hierarchy of positional embeddings (RWSE, RRWP, LPE, SPE) and show strong few-shot transfer and size extrapolation capabilities, highlighting design choices that generalize across tasks and scales. The results offer generalizable insights for GT design, training, and inference, and demonstrate that scaling and thoughtful PE selection yield robust, transferable representations across diverse graph domains. The work provides theoretical and empirical guidance for building general-purpose, scalable graph transformers with broad applicability.

Abstract

Graph Transformers (GTs) have shown strong empirical performance, yet current architectures vary widely in their use of attention mechanisms, positional embeddings (PEs), and expressivity. Existing expressivity results are often tied to specific design choices and lack comprehensive empirical validation on large-scale data. This leaves a gap between theory and practice, preventing generalizable insights that exceed particular application domains. Here, we propose the Generalized-Distance Transformer (GDT), a GT architecture using standard attention that incorporates many advancements for GTs from recent years, and develop a fine-grained understanding of the GDT's representation power in terms of attention and PEs. Through extensive experiments, we identify design choices that consistently perform well across various applications, tasks, and model scales, demonstrating strong performance in a few-shot transfer setting without fine-tuning. Our evaluation covers over eight million graphs with roughly 270M tokens across diverse domains, including image-based object detection, molecular property prediction, code summarization, and out-of-distribution algorithmic reasoning. We distill our theoretical and practical findings into several generalizable insights about effective GT design, training, and inference.

Figures (7)

• Figure 1: Overview of the GDT and accompanying evaluation. Top left: We support node- and edge-level tokenization with corresponding absolute PEs (depicted below each token). Top right: We incorporate relative PEs and edge features via the attention bias. Bottom left: We provide effective and streamlined implementations for incorporating edge features and PEs. Bottom right: All empirical evaluations are done on large-scale datasets spanning various applications.
• Figure 2: (a): Overview of our theoretical PE results in the context of existing results for PE expressivity. $A \prec B$$(A \preceq B , A \not\equiv B)$: algorithm A is strictly more powerful (at least as powerful, incomparable) than/to B (b): Trees proposed by cvetkovic1988trees used in the proof of \ref{['lemma:1WLvsRWSE']}.
• Figure 3: Results of inference-time experiments. From left to right: Few-shot transfer from Bridges to Cycles with $3$-NN over 3 random seeds; few-shot transfer from COCO to Pascal with $5$-NN over 10 random seeds; extrapolation beyond training data on MST over 3 random seeds.
• Figure 4: (a): Results on 90M and 160M models for PCQ and MST evaluated on LPE and RWSE. (b): Number of tokens evaluated per second during training for each PE. Results are obtained by averaging runtimes per token across tasks. (c): Average GPU memory requirement for each PE.
• Figure 5: A pair of CSL graphs $G_{10,2}, H_{10,3}$. We note that the path marked in blue does not exist in graph $G$ or has no replacement path.

Theorems & Definitions (69)

• Theorem 1: informal
• Theorem 2
• Proposition 3
• Proposition 4
• Definition 5: Attention (with bias)
• Definition 6: Multi-head attention (with bias)
• Definition 7: Two-layer MLP
• Definition 8: Transformer architecture
• Definition 9
• Definition 10