Relational Graph Transformer

TL;DR

RelGT proposes the first Graph Transformer architecture tailored for relational entity graphs (REGs), addressing critical challenges of heterogeneity, temporality, and schema-defined topology that limit traditional GNNs. It introduces a multi-element tokenization scheme that encodes node features, type, hop distance, time, and a local subgraph PE, enabling rich representations without heavy precomputation. A hybrid local-global Transformer attends over sampled local tokens and learnable global centroids, yielding representations that couple fine-grained structure with database-wide context. Evaluated on 21 RelBench tasks, RelGT achieves consistent improvements over GNN baselines (up to 18%), with ablations showing the essential role of subgraph PE and the nuanced benefits of the global module. This work demonstrates the viability and advantages of Graph Transformers for Relational Deep Learning, suggesting scalable pathways toward relational foundation models.

Abstract

Relational Deep Learning (RDL) is a promising approach for building state-of-the-art predictive models on multi-table relational data by representing it as a heterogeneous temporal graph. However, commonly used Graph Neural Network models suffer from fundamental limitations in capturing complex structural patterns and long-range dependencies that are inherent in relational data. While Graph Transformers have emerged as powerful alternatives to GNNs on general graphs, applying them to relational entity graphs presents unique challenges: (i) Traditional positional encodings fail to generalize to massive, heterogeneous graphs; (ii) existing architectures cannot model the temporal dynamics and schema constraints of relational data; (iii) existing tokenization schemes lose critical structural information. Here we introduce the Relational Graph Transformer (RelGT), the first graph transformer architecture designed specifically for relational tables. RelGT employs a novel multi-element tokenization strategy that decomposes each node into five components (features, type, hop distance, time, and local structure), enabling efficient encoding of heterogeneity, temporality, and topology without expensive precomputation. Our architecture combines local attention over sampled subgraphs with global attention to learnable centroids, incorporating both local and database-wide representations. Across 21 tasks from the RelBench benchmark, RelGT consistently matches or outperforms GNN baselines by up to 18%, establishing Graph Transformers as a powerful architecture for Relational Deep Learning.

Figures (5)

• Figure 1: Overview of the RelGT architecture. First, the input relational entity graph (REG) is converted into tokens where each training seed node (such as the customer node in this example) gets a fixed number of neighboring nodes, which are encoded with a multi-element tokenization strategy. These tokens are then passed through a Transformer network that builds both local and global representations, which are then fed to downstream prediction layers.
• Figure 2: The tokenization procedure. A temporal-aware subgraph sampling step extracts a fixed set of local tokens for each training seed node, denoted by the node in black. Each token incorporates its respective graph structure information, which are element-wise transformed to a common embedding space and combined to form the effective token representation to be fed to the Transformer network.
• Figure 3: The Transformer network which processes the input tokens by first building local representations using the local tokens, then incorporating global context by attending to centroids that are dynamically updated during training. The final node representations combine both local structural details and global database context, enabling effective prediction across downstream tasks.
• Figure 4: Left: Epoch runtime comparison of HGT hu2020heterogeneous and HGT+PE, with Laplacian PE (see Figure \ref{['fig:runtime-hgt']} for all tasks). The red portion shows the additional time consumed by the precomputation of Laplacian PE against the base HGT time (blue). Right: Ablation for different $K$ values as the local context size in RelGT. Results using $K = 300$ serve as the baseline (100% performance), with $K = 100$ and $K = 500$ runs measured as % of performance relative to $K=300$.
• Figure 5: Runtime Comparison of HGT and HGT+PE baseline. Adding the Laplacian Positional Encoding increases computational overhead, with penalties on average training time per epoch. The overhead for PE reaches up to 761% relative to the training time of HGT on the same dataset.