Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

What Improves the Generalization of Graph Transformers? A Theoretical Dive into the Self-attention and Positional Encoding

Hongkang Li, Meng Wang, Tengfei Ma, Sijia Liu, Zaixi Zhang, Pin-Yu Chen

TL;DR

This paper provides the first SGD-based optimization and generalization analysis for a shallow Graph Transformer with a single self-attention layer and trainable relative positional encoding in semi-supervised node classification. It introduces a data model that separates discriminative and non-discriminative patterns and a core neighborhood that governs labels, enabling a formal characterization of sample complexity and SGD iterations. Theoretical results show that larger fractions of discriminative nodes and careful graph sampling improve generalization, while self-attention induces sparse, class-focused attention and PE promotes the core neighborhood, reducing both sample complexity and training time. Empirical results on synthetic and real-world graphs corroborate the theory, with GTs(paired with PE) outperforming GCNs and GTs without PE in key regimes, offering guidance for GT design and sampling strategies in graph learning.

Abstract

Graph Transformers, which incorporate self-attention and positional encoding, have recently emerged as a powerful architecture for various graph learning tasks. Despite their impressive performance, the complex non-convex interactions across layers and the recursive graph structure have made it challenging to establish a theoretical foundation for learning and generalization. This study introduces the first theoretical investigation of a shallow Graph Transformer for semi-supervised node classification, comprising a self-attention layer with relative positional encoding and a two-layer perceptron. Focusing on a graph data model with discriminative nodes that determine node labels and non-discriminative nodes that are class-irrelevant, we characterize the sample complexity required to achieve a desirable generalization error by training with stochastic gradient descent (SGD). This paper provides the quantitative characterization of the sample complexity and number of iterations for convergence dependent on the fraction of discriminative nodes, the dominant patterns, and the initial model errors. Furthermore, we demonstrate that self-attention and positional encoding enhance generalization by making the attention map sparse and promoting the core neighborhood during training, which explains the superior feature representation of Graph Transformers. Our theoretical results are supported by empirical experiments on synthetic and real-world benchmarks.

What Improves the Generalization of Graph Transformers? A Theoretical Dive into the Self-attention and Positional Encoding

TL;DR

This paper provides the first SGD-based optimization and generalization analysis for a shallow Graph Transformer with a single self-attention layer and trainable relative positional encoding in semi-supervised node classification. It introduces a data model that separates discriminative and non-discriminative patterns and a core neighborhood that governs labels, enabling a formal characterization of sample complexity and SGD iterations. Theoretical results show that larger fractions of discriminative nodes and careful graph sampling improve generalization, while self-attention induces sparse, class-focused attention and PE promotes the core neighborhood, reducing both sample complexity and training time. Empirical results on synthetic and real-world graphs corroborate the theory, with GTs(paired with PE) outperforming GCNs and GTs without PE in key regimes, offering guidance for GT design and sampling strategies in graph learning.

Abstract

Graph Transformers, which incorporate self-attention and positional encoding, have recently emerged as a powerful architecture for various graph learning tasks. Despite their impressive performance, the complex non-convex interactions across layers and the recursive graph structure have made it challenging to establish a theoretical foundation for learning and generalization. This study introduces the first theoretical investigation of a shallow Graph Transformer for semi-supervised node classification, comprising a self-attention layer with relative positional encoding and a two-layer perceptron. Focusing on a graph data model with discriminative nodes that determine node labels and non-discriminative nodes that are class-irrelevant, we characterize the sample complexity required to achieve a desirable generalization error by training with stochastic gradient descent (SGD). This paper provides the quantitative characterization of the sample complexity and number of iterations for convergence dependent on the fraction of discriminative nodes, the dominant patterns, and the initial model errors. Furthermore, we demonstrate that self-attention and positional encoding enhance generalization by making the attention map sparse and promoting the core neighborhood during training, which explains the superior feature representation of Graph Transformers. Our theoretical results are supported by empirical experiments on synthetic and real-world benchmarks.
Paper Structure (28 sections, 12 theorems, 242 equations, 17 figures, 9 tables, 1 algorithm)

This paper contains 28 sections, 12 theorems, 242 equations, 17 figures, 9 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Works
  3. Problem Formulation and Learning Algorithm
  4. Theoretical Results
  5. Theoretical Insights
  6. Data Model Assumptions
  7. Main Theoretical Results for Graph Transformers
  8. What Does Self-Attention Improve? A Comparison with GCN
  9. How Does Positional Encoding Guide the Graph Learning Process?
  10. Proof Sketch
  11. Numerical Expriments
  12. Experiments on Synthetic Data
  13. Experiments on Real-world Dataset
  14. Conclusion, Limitation, and Future Work
  15. Additional Experiments
  16. ...and 13 more sections

Key Result

Theorem 4.4

(Generalization Guarantee of Graph Transformers) As long as for any $\epsilon\in(0,1)$, the model with $m\geq \Omega(M^2\log N)$, and the batch size $B\geq \Omega(\epsilon^{-2}\log N)$ and the number of sampled nodes $|\mathcal{S}^{n,t}|$ for each iteration $t$ larger than $\Omega(1)$. Then, after $ as long as the number of known labels satisfies where $\delta_{z_m}=\max_{n\in \mathcal{V}}|\mathc

Figures (17)

  • Figure 1: Graph Transformers in (\ref{['eqn: network']})
  • Figure 2: Example of the winning margin. Node $n$ has a non-discriminative feature ${\boldsymbol \mu}_3$ and label +1. Then $\Delta_n(1)=-2$, and $\Delta_n(2)=3$.
  • Figure 3: The impact of $\gamma_d$ and $\epsilon_\mathcal{S}$ on the sample complexity of GT.
  • Figure 4: The test Hinge loss against the number of epochs for different $\epsilon_0$.
  • Figure 5: Concentration of attention weights
  • ...and 12 more figures

Theorems & Definitions (17)

  • Definition 4.1
  • Definition 4.3
  • Theorem 4.4
  • Lemma 4.5
  • Theorem 4.6
  • Lemma 4.7
  • Theorem 4.8
  • Definition 2.2
  • Definition 2.3
  • Definition 2.4
  • ...and 7 more