FastGL: A GPU-Efficient Framework for Accelerating Sampling-Based GNN Training at Large Scale

TL;DR

This work proposes FastGL, a GPU-efficient Framework for accelerating sampling-based training of GNN at Large scale by simultaneously optimizing all above three phases, taking into account both GPU characteristics and graph structure.

Abstract

Graph Neural Networks (GNNs) have shown great superiority on non-Euclidean graph data, achieving ground-breaking performance on various graph-related tasks. As a practical solution to train GNN on large graphs with billions of nodes and edges, the sampling-based training is widely adopted by existing training frameworks. However, through an in-depth analysis, we observe that the efficiency of existing sampling-based training frameworks is still limited due to the key bottlenecks lying in all three phases of sampling-based training, i.e., subgraph sample, memory IO, and computation. To this end, we propose FastGL, a GPU-efficient Framework for accelerating sampling-based training of GNN at Large scale by simultaneously optimizing all above three phases, taking into account both GPU characteristics and graph structure. Specifically, by exploiting the inherent overlap within graph structures, FastGL develops the Match-Reorder strategy to reduce the data traffic, which accelerates the memory IO without incurring any GPU memory overhead. Additionally, FastGL leverages a Memory-Aware computation method, harnessing the GPU memory's hierarchical nature to mitigate irregular data access during computation. FastGL further incorporates the Fused-Map approach aimed at diminishing the synchronization overhead during sampling. Extensive experiments demonstrate that FastGL can achieve an average speedup of 11.8x, 2.2x and 1.5x over the state-of-the-art frameworks PyG, DGL, and GNNLab, respectively.Our code is available at https://github.com/a1bc2def6g/fastgl-ae.

Figures (16)

• Figure 1: The execution time breakdown analysis of training GCN on various graphs using different sampling-based training frameworks. More experiment setup is detailed in Section \ref{['experiment_setup']}.
• Figure 2: An example of the sampling-based training in a 2-layer GNN on the training node 0. The sampling algorithm uniformly selects two neighbors for each node.
• Figure 3: The execution time breakdown analysis on Products dataset of GCN and GIN on two NVIDIA RTX 3090 GPUs. The average time is generated from 20 epochs.
• Figure 4: Illustration of the ID map process.
• Figure 5: Overall architecture of FastGL.