On the Feasibility of Simple Transformer for Dynamic Graph Modeling

TL;DR

This work investigates using a vanilla Transformer for dynamic graph modeling by reframing dynamic graphs as sequence modeling. It introduces SimpleDyG, which constructs temporal ego-graphs and employs temporal alignment tokens to maintain a global time framework while preserving local interaction order. The approach achieves competitive or superior performance on four real-world datasets without architectural modifications to the Transformer. The findings highlight the feasibility and practicality of simple, scalable Transformer-based dynamic graph modeling with long-range dependency capture.

Abstract

Dynamic graph modeling is crucial for understanding complex structures in web graphs, spanning applications in social networks, recommender systems, and more. Most existing methods primarily emphasize structural dependencies and their temporal changes. However, these approaches often overlook detailed temporal aspects or struggle with long-term dependencies. Furthermore, many solutions overly complicate the process by emphasizing intricate module designs to capture dynamic evolutions. In this work, we harness the strength of the Transformer's self-attention mechanism, known for adeptly handling long-range dependencies in sequence modeling. Our approach offers a simple Transformer model, called SimpleDyG, tailored for dynamic graph modeling without complex modifications. We re-conceptualize dynamic graphs as a sequence modeling challenge and introduce a novel temporal alignment technique. This technique not only captures the inherent temporal evolution patterns within dynamic graphs but also streamlines the modeling process of their evolution. To evaluate the efficacy of SimpleDyG, we conduct extensive experiments on four real-world datasets from various domains. The results demonstrate the competitive performance of SimpleDyG in comparison to a series of state-of-the-art approaches despite its simple design.

Figures (5)

• Figure 1: Dynamic graph modeling can be summarized as follows: (a) Discrete-time methods treat the dynamic graph as a series of snapshots, ignoring the timing details within each. (b) Continuous-time methods factor in the timing of interactions, using them along with a graph learning process to update node representations $X_i^t$ at each time $t$. (c) Transformer-based models handle node sequences continuously, utilizing self-attention to recognize long-term dependencies.
• Figure 2: Overall framework of SimpleDyG. Best viewed in color. The numerical values adjacent to the links in (a) and (b), as well as beneath the nodes in (c), represent the time elapsed from the beginning, indicating the moments at which the links emerge (ranging from 0 to 1). The color intensity of nodes in the historical sequence represents the time span, where darker colors signify a longer duration, while lighter colors indicate a shorter duration.
• Figure 3: Performance trends of multi-step prediction.
• Figure 4: Impact of hyperparameters.
• Figure A.1: The temporal pattern of each dataset.