Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

MQ-GNN: A Multi-Queue Pipelined Architecture for Scalable and Efficient GNN Training

Irfan Ullah, Young-Koo Lee

TL;DR

MQ-GNN tackles the core bottlenecks in scalable multi-GPU GNN training by tying together a multi-queue pipelined architecture with Ready-to-Update Asynchronous Consistent Modeling (RaCoM), global neighbor sampling with caching, and an adaptive queue-sizing strategy. The method interleaves mini-batch generation, data transfer, computation, and gradient/model updates while asynchronously sharing gradients and periodically synchronizing models to control staleness. Empirical evaluations across four large-scale datasets and diverse baselines show substantial throughput gains (up to 4.6x) and improved GPU utilization (around 30%), with only minor drops in accuracy. These results demonstrate MQ-GNN as a practical, scalable framework for efficient multi-GPU GNN training in both dense and sparse graphs.

Abstract

Graph Neural Networks (GNNs) are powerful tools for learning graph-structured data, but their scalability is hindered by inefficient mini-batch generation, data transfer bottlenecks, and costly inter-GPU synchronization. Existing training frameworks fail to overlap these stages, leading to suboptimal resource utilization. This paper proposes MQ-GNN, a multi-queue pipelined framework that maximizes training efficiency by interleaving GNN training stages and optimizing resource utilization. MQ-GNN introduces Ready-to-Update Asynchronous Consistent Model (RaCoM), which enables asynchronous gradient sharing and model updates while ensuring global consistency through adaptive periodic synchronization. Additionally, it employs global neighbor sampling with caching to reduce data transfer overhead and an adaptive queue-sizing strategy to balance computation and memory efficiency. Experiments on four large-scale datasets and ten baseline models demonstrate that MQ-GNN achieves up to \boldmath $\bm{4.6\,\times}$ faster training time and 30% improved GPU utilization while maintaining competitive accuracy. These results establish MQ-GNN as a scalable and efficient solution for multi-GPU GNN training.

MQ-GNN: A Multi-Queue Pipelined Architecture for Scalable and Efficient GNN Training

TL;DR

MQ-GNN tackles the core bottlenecks in scalable multi-GPU GNN training by tying together a multi-queue pipelined architecture with Ready-to-Update Asynchronous Consistent Modeling (RaCoM), global neighbor sampling with caching, and an adaptive queue-sizing strategy. The method interleaves mini-batch generation, data transfer, computation, and gradient/model updates while asynchronously sharing gradients and periodically synchronizing models to control staleness. Empirical evaluations across four large-scale datasets and diverse baselines show substantial throughput gains (up to 4.6x) and improved GPU utilization (around 30%), with only minor drops in accuracy. These results demonstrate MQ-GNN as a practical, scalable framework for efficient multi-GPU GNN training in both dense and sparse graphs.

Abstract

Graph Neural Networks (GNNs) are powerful tools for learning graph-structured data, but their scalability is hindered by inefficient mini-batch generation, data transfer bottlenecks, and costly inter-GPU synchronization. Existing training frameworks fail to overlap these stages, leading to suboptimal resource utilization. This paper proposes MQ-GNN, a multi-queue pipelined framework that maximizes training efficiency by interleaving GNN training stages and optimizing resource utilization. MQ-GNN introduces Ready-to-Update Asynchronous Consistent Model (RaCoM), which enables asynchronous gradient sharing and model updates while ensuring global consistency through adaptive periodic synchronization. Additionally, it employs global neighbor sampling with caching to reduce data transfer overhead and an adaptive queue-sizing strategy to balance computation and memory efficiency. Experiments on four large-scale datasets and ten baseline models demonstrate that MQ-GNN achieves up to \boldmath faster training time and 30% improved GPU utilization while maintaining competitive accuracy. These results establish MQ-GNN as a scalable and efficient solution for multi-GPU GNN training.
Paper Structure (20 sections, 1 theorem, 32 equations, 1 figure, 13 tables)

This paper contains 20 sections, 1 theorem, 32 equations, 1 figure, 13 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Scalable GNN Training
  4. Graph Sampling Methods
  5. Pipeline-Based Architectures
  6. Preliminaries and Notations
  7. Node-wise Sampling (NS)
  8. Layer-wise sampling (LS)
  9. Feature Cache
  10. Multi-Queue Pipelined GNN training Architecture
  11. Determining Optimal Queue Size
  12. Evaluation
  13. Experimental Setup
  14. Comparison with Existing Systems
  15. Results and Discussion
  16. ...and 5 more sections

Key Result

Proposition 1

The optimal periodic synchronization interval $P_{ms}$, which balances computational efficiency and model consistency, is given by: This interval ensures:

Figures (1)

  • Figure 1: The GPU utilization of MQ-GCN and DGL-GCN during a single epoch of training on Reddit. Utilization is smoothed over 150 ms.

Theorems & Definitions (2)

  • Proposition 1
  • proof