Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Dynamic Graph Representation Learning via Edge Temporal States Modeling and Structure-reinforced Transformer

Shengxiang Hu, Guobing Zou, Song Yang, Shiyi Lin, Yanglan Gan, Bofeng Zhang

TL;DR

The Recurrent Structure-reinforced Graph Transformer (RSGT) is introduced, a novel framework for dynamic graph representation learning that designs a heuristic method to explicitly model edge temporal states by employing different edge types and weights based on the differences between consecutive snapshots, thereby integrating varying edge temporal states into the graph's topological structure.

Abstract

Dynamic graph representation learning has emerged as a crucial research area, driven by the growing need for analyzing time-evolving graph data in real-world applications. While recent approaches leveraging recurrent neural networks (RNNs) and graph neural networks (GNNs) have shown promise, they often fail to adequately capture the impact of temporal edge states on inter-node relationships, consequently overlooking the dynamic changes in node features induced by these evolving relationships. Furthermore, these methods suffer from GNNs' inherent over-smoothing problem, which hinders the extraction of global structural features. To address these challenges, we introduce the Recurrent Structure-reinforced Graph Transformer (RSGT), a novel framework for dynamic graph representation learning. It first designs a heuristic method to explicitly model edge temporal states by employing different edge types and weights based on the differences between consecutive snapshots, thereby integrating varying edge temporal states into the graph's topological structure. We then propose a structure-reinforced graph transformer that captures temporal node representations encoding both graph topology and evolving dynamics through a recurrent learning paradigm, enabling the extraction of both local and global structural features. Comprehensive experiments on four real-world datasets demonstrate RSGT's superior performance in discrete dynamic graph representation learning, consistently outperforming existing methods in dynamic link prediction tasks.

Dynamic Graph Representation Learning via Edge Temporal States Modeling and Structure-reinforced Transformer

TL;DR

The Recurrent Structure-reinforced Graph Transformer (RSGT) is introduced, a novel framework for dynamic graph representation learning that designs a heuristic method to explicitly model edge temporal states by employing different edge types and weights based on the differences between consecutive snapshots, thereby integrating varying edge temporal states into the graph's topological structure.

Abstract

Dynamic graph representation learning has emerged as a crucial research area, driven by the growing need for analyzing time-evolving graph data in real-world applications. While recent approaches leveraging recurrent neural networks (RNNs) and graph neural networks (GNNs) have shown promise, they often fail to adequately capture the impact of temporal edge states on inter-node relationships, consequently overlooking the dynamic changes in node features induced by these evolving relationships. Furthermore, these methods suffer from GNNs' inherent over-smoothing problem, which hinders the extraction of global structural features. To address these challenges, we introduce the Recurrent Structure-reinforced Graph Transformer (RSGT), a novel framework for dynamic graph representation learning. It first designs a heuristic method to explicitly model edge temporal states by employing different edge types and weights based on the differences between consecutive snapshots, thereby integrating varying edge temporal states into the graph's topological structure. We then propose a structure-reinforced graph transformer that captures temporal node representations encoding both graph topology and evolving dynamics through a recurrent learning paradigm, enabling the extraction of both local and global structural features. Comprehensive experiments on four real-world datasets demonstrate RSGT's superior performance in discrete dynamic graph representation learning, consistently outperforming existing methods in dynamic link prediction tasks.
Paper Structure (24 sections, 17 equations, 4 figures, 3 tables)

This paper contains 24 sections, 17 equations, 4 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Problem Formulation
  3. Approach
  4. Edge Temporal State Modeling of Dynamic Graph
  5. Recurrent Temporal Node Feature Extraction
  6. Global Semantic Encoding
  7. Structure Encoding
  8. Temporal Feature Extraction
  9. Model Training
  10. Complexity Analysis
  11. Experiments
  12. Datasets
  13. Competing Methods
  14. Experiment Results and Analyses
  15. Prediction task and parameter settings
  16. ...and 9 more sections

Figures (4)

  • Figure 1: Overall framework of the proposed Recurrent Structure-reinforced Graph Transformer (RSGT). (a) Edge Temporal State Modeling of Dynamic Graph: transforming each graph snapshot into a weighted multi-relation graph to model the edge temporal states. (b) Recurrent Temporal Node Feature Extraction: capturing both graph topology and evolving dynamics through a recurrent learning paradigm with the structure-reinforced graph transformer.
  • Figure 2: Structure-reinforced Graph Transformer for temporal node feature extraction. (a) Semantic Encoding Module: capturing global semantic correlations between nodes. (b) Structure Encoding Module: extracting topological dependencies from the weighted multi-relation difference graph, incorporating both original graph structure and temporal information. (c) Temporal Feature Extraction Module: fusing semantic and enhanced topological correlations to generate time-aware node representations.
  • Figure 3: Performance of different ablated variations of RSGT. (a)-(d) show the impact of edge temporal state modeling, while (e)-(h) illustrate the effect of graph topology learning across different performance metrics and datasets.
  • Figure 4: Impact of key hyperparameters on RSGT performance across four datasets (twi-Tennis, CollegeMsg, cit-HepTh, sx-MathOF). The subplots show the effect of (a-b) shortest path distance ($spd$), (c-d) window size, (e-f) number of encoding layers ($N_{layer}$), and (g-h) number of attention heads ($N_{head}$) on model performance. The x-axis in each subplot represents the varying values of the respective parameter, and the y-axis shows the corresponding performance metric in percentage.

Theorems & Definitions (5)

  • Definition 1: Dynamic Graph
  • Definition 2: Edge Temporal States
  • Definition 3: Dynamic Graph Representation Learning
  • Definition 4: Dynamic Link Prediction
  • Definition 5: Weighted Multi-relation Difference Graph