Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Communication-Efficient Stochastic Distributed Learning

Xiaoxing Ren, Nicola Bastianello, Karl H. Johansson, Thomas Parisini

TL;DR

A novel algorithm based on the distributed Alternating Direction Method of Multipliers (ADMM) to address the challenges of high communication costs, and large datasets is designed and it is shown that the resulting algorithm indeed converges to a stationary point, and moreover that local training accelerates convergence.

Abstract

We address distributed learning problems, both nonconvex and convex, over undirected networks. In particular, we design a novel algorithm based on the distributed Alternating Direction Method of Multipliers (ADMM) to address the challenges of high communication costs, and large datasets. Our design tackles these challenges i) by enabling the agents to perform multiple local training steps between each round of communications; and ii) by allowing the agents to employ stochastic gradients while carrying out local computations. We show that the proposed algorithm converges to a neighborhood of a stationary point, for nonconvex problems, and of an optimal point, for convex problems. We also propose a variant of the algorithm to incorporate variance reduction thus achieving exact convergence. We show that the resulting algorithm indeed converges to a stationary (or optimal) point, and moreover that local training accelerates convergence. We thoroughly compare the proposed algorithms with the state of the art, both theoretically and through numerical results.

Communication-Efficient Stochastic Distributed Learning

TL;DR

A novel algorithm based on the distributed Alternating Direction Method of Multipliers (ADMM) to address the challenges of high communication costs, and large datasets is designed and it is shown that the resulting algorithm indeed converges to a stationary point, and moreover that local training accelerates convergence.

Abstract

We address distributed learning problems, both nonconvex and convex, over undirected networks. In particular, we design a novel algorithm based on the distributed Alternating Direction Method of Multipliers (ADMM) to address the challenges of high communication costs, and large datasets. Our design tackles these challenges i) by enabling the agents to perform multiple local training steps between each round of communications; and ii) by allowing the agents to employ stochastic gradients while carrying out local computations. We show that the proposed algorithm converges to a neighborhood of a stationary point, for nonconvex problems, and of an optimal point, for convex problems. We also propose a variant of the algorithm to incorporate variance reduction thus achieving exact convergence. We show that the resulting algorithm indeed converges to a stationary (or optimal) point, and moreover that local training accelerates convergence. We thoroughly compare the proposed algorithms with the state of the art, both theoretically and through numerical results.
Paper Structure (30 sections, 10 theorems, 110 equations, 1 figure, 6 tables, 1 algorithm)

This paper contains 30 sections, 10 theorems, 110 equations, 1 figure, 6 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Comparison with the state of the art
  3. Problem Formulation and Algorithm Design
  4. Problem formulation
  5. Algorithm design
  6. Convergence Analysis and Discussion
  7. Convergence with SGD
  8. Convergence with variance reduction
  9. Discussion
  10. Choice of step-size
  11. Convergence rates
  12. Choice of variance reduction mechanism
  13. Uncoordinated parameters
  14. Numerical Results
  15. Comparison with the state of the art
  16. ...and 15 more sections

Key Result

Theorem 1

Let Assumptions as:graph, as:local-costs, and as:variance hold. If the local step-sizes satisfy $\gamma \leq \mathcal{O}(\frac{ \lambda_l}{L\tau^2 })< \gamma_{\text{sgd}}\coloneqq \min _{i=1, 2, \ldots, 6} \bar{\gamma}_i$ (see gamma_sgd in Appendix sec:preliminary_definition for the precise bound), where $x^*$ is a stationary point of eq:optimization-problem, $\lambda_u$ is the largest eigenvalue

Figures (1)

  • Figure 1: Computation time for LT-ADMM-VR to reach $\left\lVert\nabla F(\bar{x}_k)\right\rVert^2 < 10^{-7}$ for different numbers of local training epochs $\tau$.

Theorems & Definitions (17)

  • Theorem 1: Nonconvex case
  • Corollary 1: Convex case
  • Remark 1: Exact convergence
  • Theorem 2: Nonconvex case
  • Corollary 2: Convex case
  • Lemma 1
  • proof
  • Lemma 2
  • proof
  • Lemma 3
  • ...and 7 more