Grappa: Gradient-Only Communication for Scalable Graph Neural Network Training

TL;DR

Grappa tackles the communication bottleneck in distributed GNN training by enforcing gradient-only communication, training partitions in isolation, and periodically repartitioning to expose new neighborhoods. It introduces bias-corrected aggregation methods, including a batch-level estimator with shrinkage, and proves asymptotic unbiasedness under mild conditions. The approach enables phase-parallel training, achieving up to 4x average speedups (up to 13x) and maintaining or improving accuracy for deeper GNNs, while scaling to trillion-edge graphs on commodity hardware. This model-agnostic framework avoids caching and high-bandwidth interconnects, offering practical scalability for large-scale graph learning tasks.

Abstract

Cross-partition edges dominate the cost of distributed GNN training: fetching remote features and activations per iteration overwhelms the network as graphs deepen and partition counts grow. Grappa is a distributed GNN training framework that enforces gradient-only communication: during each iteration, partitions train in isolation and exchange only gradients for the global update. To recover accuracy lost to isolation, Grappa (i) periodically repartitions to expose new neighborhoods and (ii) applies a lightweight coverage-corrected gradient aggregation inspired by importance sampling. We prove the corrected estimator is asymptotically unbiased under standard support and boundedness assumptions, and we derive a batch-level variant for compatibility with common deep-learning packages that minimizes mean-squared deviation from the ideal node-level correction. We also introduce a shrinkage version that improves stability in practice. Empirical results on real and synthetic graphs show that Grappa trains GNNs 4 times faster on average (up to 13 times) than state-of-the-art systems, achieves better accuracy especially for deeper models, and sustains training at the trillion-edge scale on commodity hardware. Grappa is model-agnostic, supports full-graph and mini-batch training, and does not rely on high-bandwidth interconnects or caching.

Figures (6)

• Figure 1: Sampling-based data-parallel GNN training. Each iteration samples multi-hop neighborhoods across partitions and aggregates gradients. Following cross-partition edges triggers remote feature/activation fetches, which dominate per‑iteration cost at scale.
• Figure 2: Example of GNN training with Grappa for two super-epochs. Compared to Figure \ref{['fig:training_pipeline']}, subgraph sampling is performed without remote neighbor fetches (gradient-only communication) in each iteration and epoch. After a few epochs, we switch to a new super-epoch, creating a new partitioning layout (repartitioning).
• Figure 3: Example of efficient dynamic repartitioning via sweeping chunks. The example shows four workers operating in a data-parallel manner. The graph topology and feature data are divided into chunks $m_0$, $m_1$, $m_2$, $m_3$ and $f_0$, $f_1$, $f_2$, $f_3$ respectively. The dataset is split into chunks just once before training. The four workers then operate in parallel. Worker 0 initially loads chunks $m_0$ and $f_0$, then loads $m_1$ and $f_1$, groups these together as a partition, and trains the model on the resulting partition for a super-epoch. After that, we perform a repartition where worker 0 will load the next chunks $m_2$ and $f_2$, grouping them with $m_0$, $f_0$ to form a new partition for further training. This process repeats until all workers have seen all chunks at least once.
• Figure 4: Epoch time versus partitions for Sage-3 using Grappa and DGL (one partition per GPU). For Grappa, cross-partition communication is gradient-only and neighbor traffic is zero. Cluster: 8$\times$V100.
• Figure 5: Epoch time for Sage-3 with 16 partitions as RMAT size scales (26$\rightarrow$30). Grappa uses gradient-only communication (no neighbor traffic). 8$\times$V100 cluster.

Theorems & Definitions (4)

• theorem 1
• theorem 2
• Theorem
• Theorem