Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Full-Graph vs. Mini-Batch Training: Comprehensive Analysis from a Batch Size and Fan-Out Size Perspective

Mengfan Liu, Da Zheng, Junwei Su, Chuan Wu

TL;DR

This work systematically contrasts full-graph and mini-batch training for Graph Neural Networks from the perspectives of batch size $b$ and fan-out size $\beta$. It introduces a Wasserstein-distance-based generalization analysis to quantify how graph structure differences between training and testing data affect performance, and shows non-isotropic effects: increasing $b$ mainly influences optimization dynamics while increasing $\beta$ mainly affects generalization. Theoretical results provide concrete convergence bounds for both MSE and CE losses, and empirical studies across four real-world datasets (e.g., Reddit, ogbn-arxiv, ogbn-products, ogbn-papers100M) with GCN, GraphSAGE, and GAT validate the theory and offer practical tuning guidelines. The findings demonstrate that well-tuned mini-batch training can outperform full-graph training under memory constraints, and they supply actionable recommendations (e.g., keep $b$ below half of training nodes and $\beta$ under 15 for sparse graphs) along with hardware-agnostic iteration metrics for model selection.

Abstract

Full-graph and mini-batch Graph Neural Network (GNN) training approaches have distinct system design demands, making it crucial to choose the appropriate approach to develop. A core challenge in comparing these two GNN training approaches lies in characterizing their model performance (i.e., convergence and generalization) and computational efficiency. While a batch size has been an effective lens in analyzing such behaviors in deep neural networks (DNNs), GNNs extend this lens by introducing a fan-out size, as full-graph training can be viewed as mini-batch training with the largest possible batch size and fan-out size. However, the impact of the batch and fan-out size for GNNs remains insufficiently explored. To this end, this paper systematically compares full-graph vs. mini-batch training of GNNs through empirical and theoretical analyses from the view points of the batch size and fan-out size. Our key contributions include: 1) We provide a novel generalization analysis using the Wasserstein distance to study the impact of the graph structure, especially the fan-out size. 2) We uncover the non-isotropic effects of the batch size and the fan-out size in GNN convergence and generalization, providing practical guidance for tuning these hyperparameters under resource constraints. Finally, full-graph training does not always yield better model performance or computational efficiency than well-tuned smaller mini-batch settings. The implementation can be found in the github link: https://github.com/LIUMENGFAN-gif/GNN_fullgraph_minibatch_training.

Full-Graph vs. Mini-Batch Training: Comprehensive Analysis from a Batch Size and Fan-Out Size Perspective

TL;DR

This work systematically contrasts full-graph and mini-batch training for Graph Neural Networks from the perspectives of batch size and fan-out size . It introduces a Wasserstein-distance-based generalization analysis to quantify how graph structure differences between training and testing data affect performance, and shows non-isotropic effects: increasing mainly influences optimization dynamics while increasing mainly affects generalization. Theoretical results provide concrete convergence bounds for both MSE and CE losses, and empirical studies across four real-world datasets (e.g., Reddit, ogbn-arxiv, ogbn-products, ogbn-papers100M) with GCN, GraphSAGE, and GAT validate the theory and offer practical tuning guidelines. The findings demonstrate that well-tuned mini-batch training can outperform full-graph training under memory constraints, and they supply actionable recommendations (e.g., keep below half of training nodes and under 15 for sparse graphs) along with hardware-agnostic iteration metrics for model selection.

Abstract

Full-graph and mini-batch Graph Neural Network (GNN) training approaches have distinct system design demands, making it crucial to choose the appropriate approach to develop. A core challenge in comparing these two GNN training approaches lies in characterizing their model performance (i.e., convergence and generalization) and computational efficiency. While a batch size has been an effective lens in analyzing such behaviors in deep neural networks (DNNs), GNNs extend this lens by introducing a fan-out size, as full-graph training can be viewed as mini-batch training with the largest possible batch size and fan-out size. However, the impact of the batch and fan-out size for GNNs remains insufficiently explored. To this end, this paper systematically compares full-graph vs. mini-batch training of GNNs through empirical and theoretical analyses from the view points of the batch size and fan-out size. Our key contributions include: 1) We provide a novel generalization analysis using the Wasserstein distance to study the impact of the graph structure, especially the fan-out size. 2) We uncover the non-isotropic effects of the batch size and the fan-out size in GNN convergence and generalization, providing practical guidance for tuning these hyperparameters under resource constraints. Finally, full-graph training does not always yield better model performance or computational efficiency than well-tuned smaller mini-batch settings. The implementation can be found in the github link: https://github.com/LIUMENGFAN-gif/GNN_fullgraph_minibatch_training.
Paper Structure (144 sections, 3 theorems, 264 equations, 20 figures, 12 tables)

This paper contains 144 sections, 3 theorems, 264 equations, 20 figures, 12 tables.

Table of Contents

  1. Introduction
  2. Preliminaries
  3. Optimization Dynamic
  4. Optimization Setup
  5. Convergence Results
  6. Generalization of Mini-batch Training
  7. Problem Setup
  8. Generalization Result
  9. Empirical Study
  10. Metric: Iteration-to-accuracy
  11. Convergence
  12. Generalization
  13. Computational Efficiency
  14. Full-graph vs. Mini-batch Training after Hyperparameter Tuning
  15. Related Work
  16. ...and 129 more sections

Key Result

Theorem 1

(Convergence of Mini-batch Training with MSE) Suppose $\mathbf{W}^\text{mini}$ are generated by Gaussian initialization. Under Assumptions asmpB1. and asmpB2, if the fan-out size satisfies $C^\text{mini}_1\leq \beta\leq C^\text{mini}_2b^{\frac{3}{4}}$ for constants $C^\text{mini}_1, C^\text{mini}_2\

Figures (20)

  • Figure 1: Time-to-acc and iteration-to-acc in mini-batch and full-graph training with varying bandwidths (i.e., two inter-GPU bandwidth values: bw1=infinity > bw2=900GB/s) and computational capacities (i.e., GPU with 40GB of memory and CPU with 512GB of host memory ).
  • Figure 2: Iteration-to-loss of one-layer GraphSAGE under CE and MSE across varying learning rates and batch sizes or fan-out sizes for ogbn-products.
  • Figure 3: Test accuracy of one-layer GraphSAGE under MSE across varying learning rates and batch sizes or fan-out sizes for ogbn-products and reddit.
  • Figure 4: Iteration-to-loss of GraphSAGE under CE and MSE across varying batch and fan-out sizes.
  • Figure 5: Iteration-to-accuracy and time-to-accuracy of GraphSAGE under CE and MSE across varying batch sizes and fan-out sizes for reddit.
  • ...and 15 more figures

Theorems & Definitions (3)

  • Theorem 1
  • Theorem 2
  • Theorem 3