Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Reducing Memory Contention and I/O Congestion for Disk-based GNN Training

Qisheng Jiang, Lei Jia, Chundong Wang

TL;DR

This work targets disk-based GNN training on commodity hardware, where memory capacity and I/O bandwidth are bottlenecks. It introduces GNNDrive, a system that minimizes feature-extraction memory footprint through inter-stage buffering and eliminates I/O bottlenecks with asynchronous feature extraction and direct I/O, enabling overlapped computation and data transfer. The design supports mini-batch reordering and multi-GPU data parallelism, avoiding costly pre-processing and leveraging concurrent pipelines. Empirical results show substantial speedups over PyG+, Ginex, and MariusGNN on large graphs like Papers100M, and the approach scales to MAG240M while maintaining convergence properties. Overall, GNNDrive offers a practical, high-performance solution for disk-based GNN training on ordinary machines, broadening accessibility for researchers and small-to-medium enterprises.

Abstract

Graph neural networks (GNNs) gain wide popularity. Large graphs with high-dimensional features become common and training GNNs on them is non-trivial on an ordinary machine. Given a gigantic graph, even sample-based GNN training cannot work efficiently, since it is difficult to keep the graph's entire data in memory during the training process. Leveraging a solid-state drive (SSD) or other storage devices to extend the memory space has been studied in training GNNs. Memory and I/Os are hence critical for effectual disk-based training. We find that state-of-the-art (SoTA) disk-based GNN training systems severely suffer from issues like the memory contention between a graph's topological and feature data, and severe I/O congestion upon loading data from SSD for training. We accordingly develop GNNDrive. GNNDrive 1) minimizes the memory footprint with holistic buffer management across sampling and extracting, and 2) avoids I/O congestion through a strategy of asynchronous feature extraction. It also avoids costly data preparation on the critical path and makes the most of software and hardware resources. Experiments show that GNNDrive achieves superior performance. For example, when training with the Papers100M dataset and GraphSAGE model, GNNDrive is faster than SoTA PyG+, Ginex, and MariusGNN by 16.9x, 2.6x, and 2.7x, respectively.

Reducing Memory Contention and I/O Congestion for Disk-based GNN Training

TL;DR

This work targets disk-based GNN training on commodity hardware, where memory capacity and I/O bandwidth are bottlenecks. It introduces GNNDrive, a system that minimizes feature-extraction memory footprint through inter-stage buffering and eliminates I/O bottlenecks with asynchronous feature extraction and direct I/O, enabling overlapped computation and data transfer. The design supports mini-batch reordering and multi-GPU data parallelism, avoiding costly pre-processing and leveraging concurrent pipelines. Empirical results show substantial speedups over PyG+, Ginex, and MariusGNN on large graphs like Papers100M, and the approach scales to MAG240M while maintaining convergence properties. Overall, GNNDrive offers a practical, high-performance solution for disk-based GNN training on ordinary machines, broadening accessibility for researchers and small-to-medium enterprises.

Abstract

Graph neural networks (GNNs) gain wide popularity. Large graphs with high-dimensional features become common and training GNNs on them is non-trivial on an ordinary machine. Given a gigantic graph, even sample-based GNN training cannot work efficiently, since it is difficult to keep the graph's entire data in memory during the training process. Leveraging a solid-state drive (SSD) or other storage devices to extend the memory space has been studied in training GNNs. Memory and I/Os are hence critical for effectual disk-based training. We find that state-of-the-art (SoTA) disk-based GNN training systems severely suffer from issues like the memory contention between a graph's topological and feature data, and severe I/O congestion upon loading data from SSD for training. We accordingly develop GNNDrive. GNNDrive 1) minimizes the memory footprint with holistic buffer management across sampling and extracting, and 2) avoids I/O congestion through a strategy of asynchronous feature extraction. It also avoids costly data preparation on the critical path and makes the most of software and hardware resources. Experiments show that GNNDrive achieves superior performance. For example, when training with the Papers100M dataset and GraphSAGE model, GNNDrive is faster than SoTA PyG+, Ginex, and MariusGNN by 16.9x, 2.6x, and 2.7x, respectively.
Paper Structure (18 sections, 15 figures, 2 tables, 1 algorithm)

This paper contains 18 sections, 15 figures, 2 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Background
  3. Motivation
  4. Design of GNNDrive
  5. Overview
  6. Asynchronous Feature Extraction
  7. Reordering and Parallelism
  8. Implementation and Discussions
  9. Evaluation
  10. Training Performance
  11. Deep Dissection with GNNDrive
  12. Training Convergence
  13. A Comparison with MariusGNN
  14. Related Work
  15. Conclusion
  16. ...and 3 more sections

Figures (15)

  • Figure 1: An example of sample-based GNN training system.
  • Figure 2: Sampling time for Ginex, PyG+, and GNNDrive in varying feature dimension sizes.
  • Figure 3: CPU utilization, GPU utilization and ratio of I/O wait time with PyG+, Ginex, and MariusGNN for a time window of three epochs.
  • Figure 4: An illustration of GNNDrive's architecture.
  • Figure 5: A comparison between asynchronous and synchronous extracting.
  • ...and 10 more figures