Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

RapidGNN: Communication Efficient Large-Scale Distributed Training of Graph Neural Networks

Arefin Niam, M S Q Zulkar Nine

TL;DR

RapidGNN tackles the core bottleneck of distributed large-scale GNN training: expensive remote feature transfers. By deterministically precomputing mini-batches with fixed seeds, it enables two-stage caching and asynchronous prefetching of remote features, effectively hiding communication latency. The approach yields substantial improvements in end-to-end training throughput (average gains around 2.1× over GraphSAGE-METIS, up to 2.45× in some settings), with remote feature fetches reduced by over 4× and energy consumption cut by up to ~23%, all while preserving convergence accuracy. These results on Reddit and OGBN-Products demonstrate scalable, energy-efficient distributed GNN training with minimal changes to existing frameworks.

Abstract

Graph Neural Networks (GNNs) have achieved state-of-the-art (SOTA) performance in diverse domains. However, training GNNs on large-scale graphs poses significant challenges due to high memory demands and significant communication overhead in distributed settings. Traditional sampling-based approaches mitigate computation load to some extent but often fail to address communication inefficiencies inherent in distributed environments. This paper presents RapidGNN that introduces a deterministic sampling strategy to precompute mini-batches. By leveraging the sampling strategy, RapidGNN accurately anticipates feature access patterns, enabling optimal cache construction and timely prefetching of remote features. This reduces the frequency and latency of remote data transfers without compromising the stochastic nature of training. Evaluations on Reddit and OGBN-Products datasets demonstrate that RapidGNN achieves significant reductions in training time and remote feature fetches, outperforming existing models in both communication efficiency and throughput. Our findings highlight RapidGNN's potential for scalable, high-performance GNN training across large, real-world graph datasets along with improving energy efficiency. Our model improves end-to-end training throughput by 2.10x on average over SOTA model GraphSAGE-METIS (up to 2.45x in some settings), while cutting remote feature fetches by over 4x. It also reduces energy consumption up to 23%.

RapidGNN: Communication Efficient Large-Scale Distributed Training of Graph Neural Networks

TL;DR

RapidGNN tackles the core bottleneck of distributed large-scale GNN training: expensive remote feature transfers. By deterministically precomputing mini-batches with fixed seeds, it enables two-stage caching and asynchronous prefetching of remote features, effectively hiding communication latency. The approach yields substantial improvements in end-to-end training throughput (average gains around 2.1× over GraphSAGE-METIS, up to 2.45× in some settings), with remote feature fetches reduced by over 4× and energy consumption cut by up to ~23%, all while preserving convergence accuracy. These results on Reddit and OGBN-Products demonstrate scalable, energy-efficient distributed GNN training with minimal changes to existing frameworks.

Abstract

Graph Neural Networks (GNNs) have achieved state-of-the-art (SOTA) performance in diverse domains. However, training GNNs on large-scale graphs poses significant challenges due to high memory demands and significant communication overhead in distributed settings. Traditional sampling-based approaches mitigate computation load to some extent but often fail to address communication inefficiencies inherent in distributed environments. This paper presents RapidGNN that introduces a deterministic sampling strategy to precompute mini-batches. By leveraging the sampling strategy, RapidGNN accurately anticipates feature access patterns, enabling optimal cache construction and timely prefetching of remote features. This reduces the frequency and latency of remote data transfers without compromising the stochastic nature of training. Evaluations on Reddit and OGBN-Products datasets demonstrate that RapidGNN achieves significant reductions in training time and remote feature fetches, outperforming existing models in both communication efficiency and throughput. Our findings highlight RapidGNN's potential for scalable, high-performance GNN training across large, real-world graph datasets along with improving energy efficiency. Our model improves end-to-end training throughput by 2.10x on average over SOTA model GraphSAGE-METIS (up to 2.45x in some settings), while cutting remote feature fetches by over 4x. It also reduces energy consumption up to 23%.
Paper Structure (17 sections, 4 equations, 8 figures, 4 tables, 1 algorithm)

This paper contains 17 sections, 4 equations, 8 figures, 4 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Background and Related Work
  3. Graph Neural Networks
  4. Mini-Batch Training and Sampling in GNNs
  5. Feature Fetching in Distributed GNN Training
  6. Related Works
  7. Baseline GraphSAGE Model
  8. RapidGNN
  9. Evaluation
  10. Experimental Setup
  11. Training Time and Throughput
  12. Epoch Time Speedup
  13. Sampling + Data Copy Time
  14. Communication Reduction and Feature Reuse
  15. Resource Usage and Energy Efficiency
  16. ...and 2 more sections

Figures (8)

  • Figure 1: The working mechanism of baseline GraphSAGE hamilton2017inductive distributed training.
  • Figure 2: Epoch‑time comparison across batch sizes on OGBN‑Products (top) and Reddit (bottom)
  • Figure 3: Sampling + data copy time distributions for GCN, RapidGNN, and GraphSAGE variants.
  • Figure 4: Frequency distribution of remote feature accesses per node (midpoint of range on the x-axis). Most nodes are fetched only a handful of times, indicating a long-tail reuse pattern.
  • Figure 5: Average number of remote feature fetches per epoch versus cache size
  • ...and 3 more figures