Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Faster Convergence of Local SGD for Over-Parameterized Models

Tiancheng Qin, S. Rasoul Etesami, César A. Uribe

TL;DR

The convergence of Local SGD (or FedAvg) for such over-parameterized models in the heterogeneous data setting is analyzed and improved upon the existing literature by establishing the following convergence rates.

Abstract

Modern machine learning architectures are often highly expressive. They are usually over-parameterized and can interpolate the data by driving the empirical loss close to zero. We analyze the convergence of Local SGD (or FedAvg) for such over-parameterized models in the heterogeneous data setting and improve upon the existing literature by establishing the following convergence rates. For general convex loss functions, we establish an error bound of $Ø(1/T)$ under a mild data similarity assumption and an error bound of $Ø(K/T)$ otherwise, where $K$ is the number of local steps and $T$ is the total number of iterations. For non-convex loss functions we prove an error bound of $Ø(K/T)$. These bounds improve upon the best previous bound of $Ø(1/\sqrt{nT})$ in both cases, where $n$ is the number of nodes, when no assumption on the model being over-parameterized is made. We complete our results by providing problem instances in which our established convergence rates are tight to a constant factor with a reasonably small stepsize. Finally, we validate our theoretical results by performing large-scale numerical experiments that reveal the convergence behavior of Local SGD for practical over-parameterized deep learning models, in which the $Ø(1/T)$ convergence rate of Local SGD is clearly shown.

Faster Convergence of Local SGD for Over-Parameterized Models

TL;DR

The convergence of Local SGD (or FedAvg) for such over-parameterized models in the heterogeneous data setting is analyzed and improved upon the existing literature by establishing the following convergence rates.

Abstract

Modern machine learning architectures are often highly expressive. They are usually over-parameterized and can interpolate the data by driving the empirical loss close to zero. We analyze the convergence of Local SGD (or FedAvg) for such over-parameterized models in the heterogeneous data setting and improve upon the existing literature by establishing the following convergence rates. For general convex loss functions, we establish an error bound of under a mild data similarity assumption and an error bound of otherwise, where is the number of local steps and is the total number of iterations. For non-convex loss functions we prove an error bound of . These bounds improve upon the best previous bound of in both cases, where is the number of nodes, when no assumption on the model being over-parameterized is made. We complete our results by providing problem instances in which our established convergence rates are tight to a constant factor with a reasonably small stepsize. Finally, we validate our theoretical results by performing large-scale numerical experiments that reveal the convergence behavior of Local SGD for practical over-parameterized deep learning models, in which the convergence rate of Local SGD is clearly shown.
Paper Structure (20 sections, 13 theorems, 51 equations, 7 figures, 1 table, 1 algorithm)

This paper contains 20 sections, 13 theorems, 51 equations, 7 figures, 1 table, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Works
  3. Contributions and Organization
  4. Problem Formulation
  5. Convergence of Local SGD
  6. Convergence Rate Analysis
  7. Lower Bounds for the Convergence Rate of Local SGD
  8. Numerical Analysis
  9. ResNet18 Neural Network for Cifar10
  10. Perceptron for Linearly Separable Dataset
  11. Conclusion
  12. Proof of Theorems
  13. Preliminary Propositions
  14. Proof of Theorem \ref{['theo:convex']}
  15. Proof of Theorem \ref{['theo:non_convex']}
  16. ...and 5 more sections

Key Result

Theorem 1

Let Assumption assum:general, Assumption assum:convex and Assumption assum:interpolation (Interpolation) hold with $\mu =0$, and let $c$ be defined as in Definition def:1. Moreover, let and define $W = \sum_{t = 0}^{T-1}w_t$ and $\hat{{\mathbf x}}^T\triangleq \frac{1}{W}\sum_{i = 0}^{T-1}w_t\bar{\mathbf x}^{(t)}$. If we follow Algorithm alg:1 with stepsize $\eta \le\frac{1}{2L}$ and $K\ge 2$, the

Figures (7)

  • Figure 1: \ref{['fig:01']}: Training loss vs. communication rounds with different local steps. \ref{['fig:02']}: 1/Training loss vs. communication rounds with different local steps.
  • Figure 2: 1/(Training loss) vs. Total number of iterations $T$ with different local steps.
  • Figure 3: 1/(Training loss) vs. communication rounds for different phases. \ref{['fig:03']}: Phase 1. \ref{['fig:04']}: Transition phase. \ref{['fig:05']}: Phase 2.
  • Figure 4: Training loss vs. communication rounds with different local steps under the three data partition regimes. \ref{['fig:even_2']}: even partition regime. \ref{['fig:patho_2']}: pathological partition regime. \ref{['fig:worst_2']}: worst case partition regime.
  • Figure 5: 1/(Training loss) vs. communication rounds with different local steps under the three data partition regimes. \ref{['fig:even_1']}: even partition regime. \ref{['fig:patho_1']}: pathological partition regime. \ref{['fig:worst_1']}: worst case partition regime.
  • ...and 2 more figures

Theorems & Definitions (23)

  • Definition 1
  • Theorem 1: General convex functions
  • Theorem 2: Non-convex functions
  • Proposition 1: General Convex Functions
  • proof
  • Proposition 2: Non-convex Functions
  • proof
  • Remark 1
  • Proposition 3
  • Proposition 4
  • ...and 13 more