Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Communication-Efficient Adaptive Batch Size Strategies for Distributed Local Gradient Methods

Tim Tsz-Kit Lau, Weijian Li, Chenwei Xu, Han Liu, Mladen Kolar

Abstract

Modern deep neural networks often require distributed training with many workers due to their large size. As the number of workers increases, communication overheads become the main bottleneck in data-parallel minibatch stochastic gradient methods with per-iteration gradient synchronization. Local gradient methods like Local SGD reduce communication by only synchronizing model parameters and/or gradients after several local steps. Despite an understanding of their convergence and the importance of batch sizes for training efficiency and generalization, optimal batch sizes for local gradient methods are difficult to determine. We introduce adaptive batch size strategies for local gradient methods that increase batch sizes adaptively to reduce minibatch gradient variance. We provide convergence guarantees under homogeneous data conditions and support our claims with image classification and language modeling experiments, demonstrating the effectiveness of our strategies for both training efficiency and generalization.

Communication-Efficient Adaptive Batch Size Strategies for Distributed Local Gradient Methods

Abstract

Modern deep neural networks often require distributed training with many workers due to their large size. As the number of workers increases, communication overheads become the main bottleneck in data-parallel minibatch stochastic gradient methods with per-iteration gradient synchronization. Local gradient methods like Local SGD reduce communication by only synchronizing model parameters and/or gradients after several local steps. Despite an understanding of their convergence and the importance of batch sizes for training efficiency and generalization, optimal batch sizes for local gradient methods are difficult to determine. We introduce adaptive batch size strategies for local gradient methods that increase batch sizes adaptively to reduce minibatch gradient variance. We provide convergence guarantees under homogeneous data conditions and support our claims with image classification and language modeling experiments, demonstrating the effectiveness of our strategies for both training efficiency and generalization.
Paper Structure (39 sections, 13 theorems, 55 equations, 10 figures, 8 tables, 2 algorithms)

This paper contains 39 sections, 13 theorems, 55 equations, 10 figures, 8 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Contributions.
  3. Related Work
  4. Minibatch gradient methods.
  5. Local gradient methods.
  6. Generalization gap in large-batch training.
  7. Adaptive batch size strategies.
  8. Hyperparameter tuning for Local SGD.
  9. Preliminaries
  10. Notation.
  11. Problem Formulation
  12. Minibatch and Local Stochastic Gradient Methods
  13. Minibatch SGD.
  14. Local SGD.
  15. Adaptive Local Batch Size Strategies for Local Gradient Methods
  16. ...and 24 more sections

Key Result

Theorem 1

Suppose that ass:smoothass:strong_cvx hold with $\mu>0$. Let $(x_k^m)_{k\in\mathbb{N}, m\in\llbracket M\rrbracket}$ be the sequence of the iterates of Local SGDeqn:local_sgd with the (exact variance) local norm test eqn:exact_norm_local with $\eta_m\equiv\eta\in(0,1)$ and a constant learning rate $\ where $x_\mathsf{out}\in\{x_k^m\}_{k\in\llbracket 0,K-1\rrbracket, m\in\llbracket M\rrbracket}$ is

Figures (10)

  • Figure 1: Validation accuracy and local batch sizes of Local SHB with adaptive batch size strategies for ResNet-50 on CIFAR-10.
  • Figure 2: Validation loss and local batch sizes of Local AdamW with adaptive batch size strategies for MicroLlama 300M on C4.
  • Figure 3: Validation accuracy and local batch sizes of Local SHB with adaptive batch size strategies for ResNet-50 on CIFAR-10.
  • Figure 4: Validation accuracy and local batch sizes of Local SHB with adaptive batch size strategies for ResNet-50 on CIFAR-10 of the second seed.
  • Figure 5: Validation accuracy and local batch sizes of Local SHB with adaptive batch size strategies for ResNet-50 on CIFAR-10 of the third seed.
  • ...and 5 more figures

Theorems & Definitions (21)

  • Theorem 1: Strongly convex; $\mu>0$
  • Theorem 2: Convex; $\mu=0$
  • Theorem 3: Nonconvex
  • Remark 1: Choice of $\eta$
  • Definition B.1: $\tau$-slow sequence; Definition 10 in stich2020error
  • Lemma B.1
  • Lemma B.2
  • Lemma B.3
  • Lemma B.4: Cauchy--Schwarz inequality
  • Lemma B.5
  • ...and 11 more