Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Plain Transformers are Surprisingly Powerful Link Predictors

Quang Truong, Yu Song, Donald Loveland, Mingxuan Ju, Tong Zhao, Neil Shah, Jiliang Tang

TL;DR

Plain Transformers are shown to be surprisingly effective as link predictors when applied to fixed-budget, sampled local subgraphs. By using a standard encoder-only Transformer with a novel graph encoding and a multiplicative residual, PENCIL matches or exceeds state-of-the-art methods while requiring far fewer parameters and avoiding node identifiers or offline priors. Theoretical results connect PENCIL to established path-based scores and establish permutation-invariance in distribution, while extensive experiments demonstrate strong performance, fast convergence, and robust behavior across large-scale benchmarks. The work suggests that simple, hardware-efficient designs can capture rich graph structure for link prediction, with practical implications for deployment at scale.

Abstract

Link prediction is a core challenge in graph machine learning, demanding models that capture rich and complex topological dependencies. While Graph Neural Networks (GNNs) are the standard solution, state-of-the-art pipelines often rely on explicit structural heuristics or memory-intensive node embeddings -- approaches that struggle to generalize or scale to massive graphs. Emerging Graph Transformers (GTs) offer a potential alternative but often incur significant overhead due to complex structural encodings, hindering their applications to large-scale link prediction. We challenge these sophisticated paradigms with PENCIL, an encoder-only plain Transformer that replaces hand-crafted priors with attention over sampled local subgraphs, retaining the scalability and hardware efficiency of standard Transformers. Through experimental and theoretical analysis, we show that PENCIL extracts richer structural signals than GNNs, implicitly generalizing a broad class of heuristics and subgraph-based expressivity. Empirically, PENCIL outperforms heuristic-informed GNNs and is far more parameter-efficient than ID-embedding--based alternatives, while remaining competitive across diverse benchmarks -- even without node features. Our results challenge the prevailing reliance on complex engineering techniques, demonstrating that simple design choices are potentially sufficient to achieve the same capabilities.

Plain Transformers are Surprisingly Powerful Link Predictors

TL;DR

Plain Transformers are shown to be surprisingly effective as link predictors when applied to fixed-budget, sampled local subgraphs. By using a standard encoder-only Transformer with a novel graph encoding and a multiplicative residual, PENCIL matches or exceeds state-of-the-art methods while requiring far fewer parameters and avoiding node identifiers or offline priors. Theoretical results connect PENCIL to established path-based scores and establish permutation-invariance in distribution, while extensive experiments demonstrate strong performance, fast convergence, and robust behavior across large-scale benchmarks. The work suggests that simple, hardware-efficient designs can capture rich graph structure for link prediction, with practical implications for deployment at scale.

Abstract

Link prediction is a core challenge in graph machine learning, demanding models that capture rich and complex topological dependencies. While Graph Neural Networks (GNNs) are the standard solution, state-of-the-art pipelines often rely on explicit structural heuristics or memory-intensive node embeddings -- approaches that struggle to generalize or scale to massive graphs. Emerging Graph Transformers (GTs) offer a potential alternative but often incur significant overhead due to complex structural encodings, hindering their applications to large-scale link prediction. We challenge these sophisticated paradigms with PENCIL, an encoder-only plain Transformer that replaces hand-crafted priors with attention over sampled local subgraphs, retaining the scalability and hardware efficiency of standard Transformers. Through experimental and theoretical analysis, we show that PENCIL extracts richer structural signals than GNNs, implicitly generalizing a broad class of heuristics and subgraph-based expressivity. Empirically, PENCIL outperforms heuristic-informed GNNs and is far more parameter-efficient than ID-embedding--based alternatives, while remaining competitive across diverse benchmarks -- even without node features. Our results challenge the prevailing reliance on complex engineering techniques, demonstrating that simple design choices are potentially sufficient to achieve the same capabilities.
Paper Structure (39 sections, 9 theorems, 32 equations, 8 figures, 10 tables)

This paper contains 39 sections, 9 theorems, 32 equations, 8 figures, 10 tables.

Table of Contents

  1. Introduction
  2. Preliminaries
  3. Plain Transformers As Link Predictors
  4. Graph Encoding Scheme
  5. Architecture Design
  6. Estimating Pairwise Heuristics with PENCIL
  7. Theoretical Analysis
  8. Distributional Permutation Invariance
  9. What Makes PENCIL a Strong Link Predictor?
  10. Expressive Power of Plain Transformers on Link Prediction
  11. Experimental Results
  12. Main Results
  13. Effects of Model Depth
  14. Related Work
  15. Conclusion, Limitations and Future Work
  16. ...and 24 more sections

Key Result

Theorem 4.1

Let $f$ be any deterministic measurable function. Define a randomized predictor $S(\mathbf{A}; u, v) := f(\mathbf{P}_\rho \mathbf{A} \mathbf{P}_\rho^\top)$, where $\rho$ is drawn uniformly from the set of permutations satisfying $\rho(u)=0$ and $\rho(v)=1$. Then for any node relabeling $\pi$, the ou where $\mathbf{A}' = \mathbf{P}_\pi \mathbf{A} \mathbf{P}_\pi^\top$, $u' = \pi(u)$, and $v' = \pi(v

Figures (8)

  • Figure 1: Parameter efficiency vs. performance on ogbl-ppa. PENCIL (black$\bigstar$) achieves state-of-the-art performance without node IDs or handcrafted heuristics, using orders of magnitude fewer parameters than leading ID-based methods.
  • Figure 2: Visualization of the input encoding scheme for a sampled subgraph revolving around a query link (dashed), with the number of nodes $N=5$ and the sampling budget $N_{\max}=6$. Blue block corresponds to the sampled nodes (the context set). Green block contains one-hot identifiers of nodes, yellow block contains nodes' adjacency row, purple block contains role flags, and red block contains task tokens. Context-node ordering is permutable; only $v_{\mathrm{src}}$ and $v_{\mathrm{dst}}$ are fixed to $v_0$ and $v_1$.
  • Figure 3: PENCIL architecture. Node features can be optionally used as discussed in Section \ref{['sec:architectural-details']}, but we omit them from the figure for clarity.
  • Figure 4: RMSE of PENCIL and other GNNs for estimating pairwise heuristics on the cora dataset.
  • Figure 5: Effect of model depth on different performance metrics across the cora, pubmed, and ogbl-collab datasets.
  • ...and 3 more figures

Theorems & Definitions (20)

  • Theorem 4.1
  • Proposition 4.2
  • Corollary 4.3
  • Remark 4.4
  • Proposition 4.5
  • Definition 4.6: Mutual coherence
  • Definition 4.7: The Welch bound
  • Proposition 4.8
  • Theorem 4.9
  • Theorem 2.1
  • ...and 10 more