Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Asynchronous Decentralized SGD under Non-Convexity: A Block-Coordinate Descent Framework

Yijie Zhou, Shi Pu

TL;DR

This work addresses decentralized non-convex optimization under bounded computation and communication delays by introducing ADSGD and linking its convergence to ASBCD. It proves that ASBCD with stochastic gradients achieves a rate of $O(1/\sqrt{K})$ for non-convex objectives, and that ADSGD converges with computation-delay-independent step sizes via a double-step-size mechanism on the augmented function $L_\alpha(\mathbf{x})$, yielding a rate on stationary points affected by consensus error ($O(1/K^{1/3})$ in the provided bounds). The authors demonstrate substantial practical benefits: reduced per-iteration communication and memory, no reliance on data heterogeneity bounds, and strong resilience to delays, backed by empirical results on non-convex logistic regression and VGG11 training showing faster wall-clock convergence under stragglers. The proposed framework offers a simple, scalable, and delay-robust approach for real-world decentralized learning tasks.

Abstract

Decentralized optimization has become vital for leveraging distributed data without central control, enhancing scalability and privacy. However, practical deployments face fundamental challenges due to heterogeneous computation speeds and unpredictable communication delays. This paper introduces a refined model of Asynchronous Decentralized Stochastic Gradient Descent (ADSGD) under practical assumptions of bounded computation and communication times. To understand the convergence of ADSGD, we first analyze Asynchronous Stochastic Block Coordinate Descent (ASBCD) as a tool, and then show that ADSGD converges under computation-delay-independent step sizes. The convergence result is established without assuming bounded data heterogeneity. Empirical experiments reveal that ADSGD outperforms existing methods in wall-clock convergence time across various scenarios. With its simplicity, efficiency in memory and communication, and resilience to communication and computation delays, ADSGD is well-suited for real-world decentralized learning tasks.

Asynchronous Decentralized SGD under Non-Convexity: A Block-Coordinate Descent Framework

TL;DR

This work addresses decentralized non-convex optimization under bounded computation and communication delays by introducing ADSGD and linking its convergence to ASBCD. It proves that ASBCD with stochastic gradients achieves a rate of for non-convex objectives, and that ADSGD converges with computation-delay-independent step sizes via a double-step-size mechanism on the augmented function , yielding a rate on stationary points affected by consensus error ( in the provided bounds). The authors demonstrate substantial practical benefits: reduced per-iteration communication and memory, no reliance on data heterogeneity bounds, and strong resilience to delays, backed by empirical results on non-convex logistic regression and VGG11 training showing faster wall-clock convergence under stragglers. The proposed framework offers a simple, scalable, and delay-robust approach for real-world decentralized learning tasks.

Abstract

Decentralized optimization has become vital for leveraging distributed data without central control, enhancing scalability and privacy. However, practical deployments face fundamental challenges due to heterogeneous computation speeds and unpredictable communication delays. This paper introduces a refined model of Asynchronous Decentralized Stochastic Gradient Descent (ADSGD) under practical assumptions of bounded computation and communication times. To understand the convergence of ADSGD, we first analyze Asynchronous Stochastic Block Coordinate Descent (ASBCD) as a tool, and then show that ADSGD converges under computation-delay-independent step sizes. The convergence result is established without assuming bounded data heterogeneity. Empirical experiments reveal that ADSGD outperforms existing methods in wall-clock convergence time across various scenarios. With its simplicity, efficiency in memory and communication, and resilience to communication and computation delays, ADSGD is well-suited for real-world decentralized learning tasks.
Paper Structure (30 sections, 6 theorems, 47 equations, 16 figures, 2 tables, 3 algorithms)

This paper contains 30 sections, 6 theorems, 47 equations, 16 figures, 2 tables, 3 algorithms.

Table of Contents

  1. Introduction
  2. Related Work
  3. Main Contributions
  4. The algorithms
  5. ASBCD
  6. ADSGD
  7. Notation
  8. Convergence Analysis
  9. Assumptions
  10. Convergence of ASBCD
  11. Convergence of ADSGD
  12. Empirical Evaluation
  13. Non-convex Logistic Regression on MNIST
  14. VGG11 on CIFAR-10
  15. Scalability
  16. ...and 15 more sections

Key Result

Lemma 3.8

For problem eq:optimization, given Assumption asm:b-l_bounded-smooth - asm:partialasynchrony, and $\alpha < \frac{1}{(D + 1/2)L}$, the sequence $\{\mathbf{x}^k\}$ generated by (eq:ABCD_update) satisfies the following relation: where $C_0 = D^2+3B^2(D + 2D^3)$.

Figures (16)

  • Figure 1: Schematics of $s_{ij}^k$ for ASBCD (left) and ADSGD (right), with the computation and communication of primary focus highlighted in a darker shade.
  • Figure 2: Loss plot of Logistic Regression on MNIST. Case 5 is excluded for clarity. See the appendix for details.
  • Figure 3: Loss plot of VGG on CIFAR10. Case 5 is excluded for clarity. See the appendix for details.
  • Figure 4: Speedup of ADSGD w.r.t. number of agents under a ring topology.
  • Figure 5: Logistic Regression - No Straggler - Training Loss - $\zeta=1$
  • ...and 11 more figures

Theorems & Definitions (15)

  • Remark 2.1
  • Remark 2.2
  • Remark 3.4
  • Lemma 3.8
  • Theorem 3.9
  • Remark 3.10
  • Remark 3.11
  • Corollary 3.12
  • Lemma C.1
  • proof
  • ...and 5 more