DHIL-GT: Scalable Graph Transformer with Decoupled Hierarchy Labeling

TL;DR

DHIL-GT addresses the scalability bottleneck of Graph Transformers by decoupling graph computation from learning through a PLL-based hierarchical label graph. It precomputes fixed-size subgraph tokens and SPD-based positional encodings, enabling end-to-end Transformer learning with minimal graph-scale computations and improved mini-batch efficiency. The approach introduces subgraph tokens, SPD bias, and a virtual node to capture hierarchical connectivity and handle heterophily, achieving competitive accuracy on homophilous and heterophilous graphs while significantly reducing computation and memory overhead. Theoretical complexity highlights linear scaling in graph size for precomputation and learning, and empirical results demonstrate substantial speedups and effective performance on large-scale benchmarks, making Graph Transformers more practical for real-world graphs.

Abstract

Graph Transformer (GT) has recently emerged as a promising neural network architecture for learning graph-structured data. However, its global attention mechanism with quadratic complexity concerning the graph scale prevents wider application to large graphs. While current methods attempt to enhance GT scalability by altering model architecture or encoding hierarchical graph data, our analysis reveals that these models still suffer from the computational bottleneck related to graph-scale operations. In this work, we target the GT scalability issue and propose DHIL-GT, a scalable Graph Transformer that simplifies network learning by fully decoupling the graph computation to a separate stage in advance. DHIL-GT effectively retrieves hierarchical information by exploiting the graph labeling technique, as we show that the graph label hierarchy is more informative than plain adjacency by offering global connections while promoting locality, and is particularly suitable for handling complex graph patterns such as heterophily. We further design subgraph sampling and positional encoding schemes for precomputing model input on top of graph labels in an end-to-end manner. The training stage thus favorably removes graph-related computations, leading to ideal mini-batch capability and GPU utilization. Notably, the precomputation and training processes of DHIL-GT achieve complexities linear to the number of graph edges and nodes, respectively. Extensive experiments demonstrate that DHIL-GT is efficient in terms of computational boost and mini-batch capability over existing scalable Graph Transformer designs on large-scale benchmarks, while achieving top-tier effectiveness on both homophilous and heterophilous graphs.

Figures (5)

• Figure 1: Visualization of the hierarchy of (a) the original graph $\mathcal{G}$ and (b) the label graph $\Hat{\mathcal{G}}$ on the heterophilous dataset chameleon. Color of each node denotes its class.
• Figure 2: Examples of three properties of the label graph $\Hat{\mathcal{G}}$ corresponding to the original graph $\mathcal{G}$. Number inside each node denotes its index in descending order of node degrees.
• Figure 3: High-level framework of DHIL-GT including two consecutive stages of precomputation and Transformer learning. The precomputation stage processes input graph into the label graph, which is then used to generate the subgraph structure and SPD bias. During training, the subgraph tokens are applied as input features for each node, while SPD is regarded as positional encoding for Transformer layers. A readout block aggregates the subgraph representation to the ego node for prediction.
• Figure 4: Effect of subgraph parameters on chameleon.
• Figure 5: Visualization of the hierarchy of original and label graphs on realistic datasets. Color of each node denotes its class.

Theorems & Definitions (1)

• corollary 1: akiba2013