Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

LPFormer: An Adaptive Graph Transformer for Link Prediction

Harry Shomer, Yao Ma, Haitao Mao, Juanhui Li, Bo Wu, Jiliang Tang

TL;DR

LPFormer is proposed, which attempts to adaptively learn the pairwise encodings for each link via an attention module that learns the pairwise encoding that exists between nodes by modeling multiple factors integral to link prediction.

Abstract

Link prediction is a common task on graph-structured data that has seen applications in a variety of domains. Classically, hand-crafted heuristics were used for this task. Heuristic measures are chosen such that they correlate well with the underlying factors related to link formation. In recent years, a new class of methods has emerged that combines the advantages of message-passing neural networks (MPNN) and heuristics methods. These methods perform predictions by using the output of an MPNN in conjunction with a "pairwise encoding" that captures the relationship between nodes in the candidate link. They have been shown to achieve strong performance on numerous datasets. However, current pairwise encodings often contain a strong inductive bias, using the same underlying factors to classify all links. This limits the ability of existing methods to learn how to properly classify a variety of different links that may form from different factors. To address this limitation, we propose a new method, LPFormer, which attempts to adaptively learn the pairwise encodings for each link. LPFormer models the link factors via an attention module that learns the pairwise encoding that exists between nodes by modeling multiple factors integral to link prediction. Extensive experiments demonstrate that LPFormer can achieve SOTA performance on numerous datasets while maintaining efficiency. The code is available at The code is available at https://github.com/HarryShomer/LPFormer.

LPFormer: An Adaptive Graph Transformer for Link Prediction

TL;DR

LPFormer is proposed, which attempts to adaptively learn the pairwise encodings for each link via an attention module that learns the pairwise encoding that exists between nodes by modeling multiple factors integral to link prediction.

Abstract

Link prediction is a common task on graph-structured data that has seen applications in a variety of domains. Classically, hand-crafted heuristics were used for this task. Heuristic measures are chosen such that they correlate well with the underlying factors related to link formation. In recent years, a new class of methods has emerged that combines the advantages of message-passing neural networks (MPNN) and heuristics methods. These methods perform predictions by using the output of an MPNN in conjunction with a "pairwise encoding" that captures the relationship between nodes in the candidate link. They have been shown to achieve strong performance on numerous datasets. However, current pairwise encodings often contain a strong inductive bias, using the same underlying factors to classify all links. This limits the ability of existing methods to learn how to properly classify a variety of different links that may form from different factors. To address this limitation, we propose a new method, LPFormer, which attempts to adaptively learn the pairwise encodings for each link. LPFormer models the link factors via an attention module that learns the pairwise encoding that exists between nodes by modeling multiple factors integral to link prediction. Extensive experiments demonstrate that LPFormer can achieve SOTA performance on numerous datasets while maintaining efficiency. The code is available at The code is available at https://github.com/HarryShomer/LPFormer.
Paper Structure (31 sections, 2 theorems, 35 equations, 5 figures, 7 tables, 1 algorithm)

This paper contains 31 sections, 2 theorems, 35 equations, 5 figures, 7 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Background
  3. Related Work
  4. Preliminaries
  5. The Proposed Framework
  6. A General View of Pairwise Encodings
  7. Modeling Pairwise Encodings via Attention
  8. PPR-Based Relative Positional Encodings
  9. Efficiently Attending to the Graph Context
  10. LPFormer
  11. Experiments
  12. Experimental Settings
  13. Main Results
  14. Performance by LP Factor
  15. Ablation Study
  16. ...and 16 more sections

Key Result

Proposition 1

Consider a target link $(a, b)$ and a node $u \in \mathcal{V} \setminus \{a, b\}$. The PPR pagerank score of a root node $i$ and target node $j$ with teleportation probability $\alpha$ is denoted by $\text{ppr}(i, j)$. Let $r_{a}^k (u)$ be the probability of a walk of length $k$ beginning at node $a

Figures (5)

  • Figure 1: Example of multiple heuristic scores for the candidate links (source, 5), (source, 6), and (source, 7). Each heuristic corresponds to a different LP factor -- local (CNs), global (Katz), and feature proximity (Feat-Sim).
  • Figure 2: An overview of LPFormer. (1) Encode the nodes via a MPNN. (2) For a given target link, we determine which nodes to attend to ($\hat{\mathcal{N}}(a, b)$) via the PPR-based thresholding technique in Eq. \ref{['eq:ppr_threshold']}. (3) The pairwise encoding is computed by attending to each node, $u \in \hat{\mathcal{N}}(a, b)$ using the feature and relative positional encoding $\mathbf{\text{rpe}}_{(a, b, u)}$. (4) The pairwise encoding, node representations, and counts of different node types are concatenated and used to compute the final probability of the target link existing.
  • Figure 3: Performance on links that contain one dominant LP factor. Results are on (a) Cora, (b) Citeseer, and (c) ogbl-collab.
  • Figure 4: Comparison of training time of 1 epoch between LPFormer and NCNC. The mean degree is in parentheses.
  • Figure 5: Performance for target links when there is only one LP factor strongly expressed. Results are on (a) Pubmed, (b) ogbl-ppa. We note that due the quality of features used, we omit the feature proximity factor for ogbl-ppa from our analysis

Theorems & Definitions (2)

  • Proposition 1
  • Proposition 1