ReInc: Scaling Training of Dynamic Graph Neural Networks

TL;DR

ReInc tackles the heavy cost of training dynamic graph neural networks on large graphs by introducing three core ideas: reuse opportunities across DGNN components, incremental aggregation using delta graphs with $Agg_t = Agg_{t-1} - F_{t-1} *_{aggr} G^-_t + F_t *_{aggr} G^+_t$, and a DGNN-aware two-level cache plus a consecutive-block distributed training strategy. It supports both stacked and integrated DGNNs and introduces a seq-first mini-batch training scheme to maximize cache hits and minimize memory usage. A prototype on DGL/PyTorch demonstrates up to an order-of-magnitude speedup over state-of-the-art systems while maintaining correctness, with favorable scaling across machines, feature/hiddens, and sequence lengths. This work enables practical, scalable DGNN training for real-world, large-scale dynamic graphs without restrictive assumptions about dynamicity or graph size.

Abstract

Dynamic Graph Neural Networks (DGNNs) have gained widespread attention due to their applicability in diverse domains such as traffic network prediction, epidemiological forecasting, and social network analysis. In this paper, we present ReInc, a system designed to enable efficient and scalable training of DGNNs on large-scale graphs. ReInc introduces key innovations that capitalize on the unique combination of Graph Neural Networks (GNNs) and Recurrent Neural Networks (RNNs) inherent in DGNNs. By reusing intermediate results and incrementally computing aggregations across consecutive graph snapshots, ReInc significantly enhances computational efficiency. To support these optimizations, ReInc incorporates a novel two-level caching mechanism with a specialized caching policy aligned to the DGNN execution workflow. Additionally, ReInc addresses the challenges of managing structural and temporal dependencies in dynamic graphs through a new distributed training strategy. This approach eliminates communication overheads associated with accessing remote features and redistributing intermediate results. Experimental results demonstrate that ReInc achieves up to an order of magnitude speedup compared to state-of-the-art frameworks, tested across various dynamic GNN architectures and real-world graph datasets.

Figures (14)

• Figure 1: (a) A stacked DGNN where an RNN takes the output of a GNN as its input. (b) An integrated DGNN (GraphRNN) that replaces matrix multiplications in RNN gates with GNNs.
• Figure 2: Sliding window mechanism to generate sequences in DGNNs. Consecutive sequences overlap with each other.
• Figure 3: Sequence-to-sequence computation in DGNNs. For each input sequence, the computation of one snapshot $G_t$ depends on the hidden output of the previous snapshot $h_{t-1}$.
• Figure 4: Different graph partitioning schemes for a dynamic graph with $T$ snapshots across $M$ machines. Here, the sequence length is $3$, and each color (or row) represents a node. Features and topology for a node are partitioned on the same machine for all schemes.
• Figure 5: ReInc extracts two delta graphs $G^-_t$ (edge deletions) and $G^+_t$ (edge insertions) and computes aggregation results incrementally to eliminate redundant computation.