Simple and Efficient Heterogeneous Temporal Graph Neural Network

TL;DR

This work innovatively integrate temporal modeling into spatial learning via a novel dynamic attention mechanism, which retains attention information from historical graph snapshots to guide subsequent attention computation, thereby improving the overall discriminative representations learning of HTGs.

Abstract

Heterogeneous temporal graphs (HTGs) are ubiquitous data structures in the real world. Recently, to enhance representation learning on HTGs, numerous attention-based neural networks have been proposed. Despite these successes, existing methods rely on a decoupled temporal and spatial learning paradigm, which weakens interactions of spatio-temporal information and leads to a high model complexity. To bridge this gap, we propose a novel learning paradigm for HTGs called Simple and Efficient Heterogeneous Temporal Graph N}eural Network (SE-HTGNN). Specifically, we innovatively integrate temporal modeling into spatial learning via a novel dynamic attention mechanism, which retains attention information from historical graph snapshots to guide subsequent attention computation, thereby improving the overall discriminative representations learning of HTGs. Additionally, to comprehensively and adaptively understand HTGs, we leverage large language models to prompt SE-HTGNN, enabling the model to capture the implicit properties of node types as prior knowledge. Extensive experiments demonstrate that SE-HTGNN achieves up to 10x speed-up over the state-of-the-art and latest baseline while maintaining the best forecasting accuracy.

Figures (6)

• Figure 1: (a) shows a heterogeneous temporal graph (HTG). (b) shows a general framework of attention-based HDGNNs. However, decoupled spatial-temporal modeling strategy results in attention discontinuity (attention on each graph snapshot is computed in isolation), increasing the risk of convergence to local optima.
• Figure 2: (a) The overall architecture of SE-HTGNN. (b) The LLM-enhanced prompt module takes type prompts as input and generates initial attention coefficients $e^0$ by leveraging LLM-enhanced prior knowledge. (c) The dynamic-attention-based graph learning module predicts current attention coefficients $e^{T}$ from current features and historical coefficients $e^{T-1}$, enabling effective representations fusion. (d) The linear project module maps representations to future prediction steps for downstream tasks.
• Figure 3: Comparison of convergence speed.
• Figure 4: Hyper-parameter sensitive study.
• Figure 5: Visualization on Covid-19 dataset.