Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Temporal Graph Pattern Machine

Yijun Ma, Zehong Wang, Weixiang Sun, Yanfang Ye

TL;DR

Temporal Graph Pattern Machine (TGPM) tackles learning generalizable temporal evolution mechanisms in dynamic graphs, moving beyond task-specific representations. It builds interaction patches from temporally biased random walks and encodes them with a Transformer backbone, augmented by target-relative time encodings. Two self-supervised objectives, Masked Token Modeling and Next Time Prediction, enforce learning of multi-scale temporal dependencies and future timing, enabling robust cross-domain transfer. Empirical results on Enron, ICEWS1819, and Googlemap CT show state-of-the-art performance for transductive and inductive link prediction and strong transferability, while revealing sensitivity to highly homogeneous temporal burstiness and suggesting meta-pattern aggregation as a potential fix.

Abstract

Temporal graph learning is pivotal for deciphering dynamic systems, where the core challenge lies in explicitly modeling the underlying evolving patterns that govern network transformation. However, prevailing methods are predominantly task-centric and rely on restrictive assumptions -- such as short-term dependency modeling, static neighborhood semantics, and retrospective time usage. These constraints hinder the discovery of transferable temporal evolution mechanisms. To address this, we propose the Temporal Graph Pattern Machine (TGPM), a foundation framework that shifts the focus toward directly learning generalized evolving patterns. TGPM conceptualizes each interaction as an interaction patch synthesized via temporally-biased random walks, thereby capturing multi-scale structural semantics and long-range dependencies that extend beyond immediate neighborhoods. These patches are processed by a Transformer-based backbone designed to capture global temporal regularities while adapting to context-specific interaction dynamics. To further empower the model, we introduce a suite of self-supervised pre-training tasks -- specifically masked token modeling and next-time prediction -- to explicitly encode the fundamental laws of network evolution. Extensive experiments show that TGPM consistently achieves state-of-the-art performance in both transductive and inductive link prediction, demonstrating exceptional cross-domain transferability.

Temporal Graph Pattern Machine

TL;DR

Temporal Graph Pattern Machine (TGPM) tackles learning generalizable temporal evolution mechanisms in dynamic graphs, moving beyond task-specific representations. It builds interaction patches from temporally biased random walks and encodes them with a Transformer backbone, augmented by target-relative time encodings. Two self-supervised objectives, Masked Token Modeling and Next Time Prediction, enforce learning of multi-scale temporal dependencies and future timing, enabling robust cross-domain transfer. Empirical results on Enron, ICEWS1819, and Googlemap CT show state-of-the-art performance for transductive and inductive link prediction and strong transferability, while revealing sensitivity to highly homogeneous temporal burstiness and suggesting meta-pattern aggregation as a potential fix.

Abstract

Temporal graph learning is pivotal for deciphering dynamic systems, where the core challenge lies in explicitly modeling the underlying evolving patterns that govern network transformation. However, prevailing methods are predominantly task-centric and rely on restrictive assumptions -- such as short-term dependency modeling, static neighborhood semantics, and retrospective time usage. These constraints hinder the discovery of transferable temporal evolution mechanisms. To address this, we propose the Temporal Graph Pattern Machine (TGPM), a foundation framework that shifts the focus toward directly learning generalized evolving patterns. TGPM conceptualizes each interaction as an interaction patch synthesized via temporally-biased random walks, thereby capturing multi-scale structural semantics and long-range dependencies that extend beyond immediate neighborhoods. These patches are processed by a Transformer-based backbone designed to capture global temporal regularities while adapting to context-specific interaction dynamics. To further empower the model, we introduce a suite of self-supervised pre-training tasks -- specifically masked token modeling and next-time prediction -- to explicitly encode the fundamental laws of network evolution. Extensive experiments show that TGPM consistently achieves state-of-the-art performance in both transductive and inductive link prediction, demonstrating exceptional cross-domain transferability.
Paper Structure (21 sections, 5 theorems, 23 equations, 4 figures, 5 tables)

This paper contains 21 sections, 5 theorems, 23 equations, 4 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Temporal Graph Pattern Machine
  3. Temporal Context Representation
  4. TGPM Backbone
  5. TGPM Pre-Training
  6. Experiments
  7. Experimental Settings
  8. In-Domain Performance Comparison
  9. Cross-Domain Transferability
  10. Scalability Analysis
  11. Ablation Study
  12. Failure Case Analysis
  13. Related Work
  14. Conclusion
  15. Additional Experimental Details
  16. ...and 6 more sections

Key Result

Proposition 2.1

Given a temporal random walk $w=(v_0,v_1,\dots,v_L)$ with $\mathcal{T}(v_i,v_{i+1})<t', i=0,1,\dots,L-1$ and a causal walk $w_c = (u_0,u_1,\dots,u_L)$ with $\mathcal{T}(u_{i-1},u_i) < \mathcal{T}(u_i, u_{i+1}) < t', i=1,2,\dots,L-1$, $w$ can capture more complex evolving semantics than $w_c$.

Figures (4)

  • Figure 1: Overview of TGPM. (a) Temporal context of an interaction is represented by an interaction patch aggregated from a set of temporally biased random walks. (b) Transformer-based TGPM encoder adapt local structural and temporal semantics to target-specific temporal context, capturing global temporal regularities. (c) MTM and NTP pre-training enforce TGPM to learn from multi-scale temporal dynamics and the temporal rhythms of evolving patterns.
  • Figure 2: Comparison with self-supervised methods over cross-domain transferability. TGPM achieves the strongest performance compared to prior works across all datasets.
  • Figure 3: Model scalability behavior. Increasing model parameters can consistently enhance performance on transductive settings, showing the great potential of TGPM acting as the scalable backbone in temporal graph learning.
  • Figure 4: Failure Case Analysis in Enron, which has large-scale homogeneous temporal burstiness, leading to highly similar sampled patterns and performance degradation of pretraining.

Theorems & Definitions (13)

  • Proposition 2.1
  • Proposition 2.2
  • Definition 2.1: Temporal Random Walk Expressiveness
  • Proposition 2.2: Expressiveness of Non-Monotonic Temporal Walks
  • proof
  • Definition 2.3: Temporal Reachability Gap
  • Remark 2.4
  • Definition 2.5: $r$-Markov Dependency
  • Definition 2.6: Predictive Information
  • Proposition 2.7: Block Size Determines Learnable Dependency Scale
  • ...and 3 more