Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Asynchronous Distributed Optimization with Delay-free Parameters

Xuyang Wu, Changxin Liu, Sindri Magnusson, Mikael Johansson

TL;DR

Asynchronous versions of two distributed algorithms, Prox-DGD and DGD-ATC, for solving consensus optimization problems over undirected networks are developed, establishing convergence guarantees for convex and strongly convex problems under both partial and total asynchrony.

Abstract

Existing asynchronous distributed optimization algorithms often use diminishing step-sizes that cause slow practical convergence, or use fixed step-sizes that depend on and decrease with an upper bound of the delays. Not only are such delay bounds hard to obtain in advance, but they also tend to be large and rarely attained, resulting in unnecessarily slow convergence. This paper develops asynchronous versions of two distributed algorithms, Prox-DGD and DGD-ATC, for solving consensus optimization problems over undirected networks. In contrast to alternatives, our algorithms can converge to the fixed point set of their synchronous counterparts using step-sizes that are independent of the delays. We establish convergence guarantees for convex and strongly convex problems under both partial and total asynchrony. We also show that the convergence speed of the two asynchronous methods adjusts to the actual level of asynchrony rather than being constrained by the worst-case. Numerical experiments demonstrate a strong practical performance of our asynchronous algorithms.

Asynchronous Distributed Optimization with Delay-free Parameters

TL;DR

Asynchronous versions of two distributed algorithms, Prox-DGD and DGD-ATC, for solving consensus optimization problems over undirected networks are developed, establishing convergence guarantees for convex and strongly convex problems under both partial and total asynchrony.

Abstract

Existing asynchronous distributed optimization algorithms often use diminishing step-sizes that cause slow practical convergence, or use fixed step-sizes that depend on and decrease with an upper bound of the delays. Not only are such delay bounds hard to obtain in advance, but they also tend to be large and rarely attained, resulting in unnecessarily slow convergence. This paper develops asynchronous versions of two distributed algorithms, Prox-DGD and DGD-ATC, for solving consensus optimization problems over undirected networks. In contrast to alternatives, our algorithms can converge to the fixed point set of their synchronous counterparts using step-sizes that are independent of the delays. We establish convergence guarantees for convex and strongly convex problems under both partial and total asynchrony. We also show that the convergence speed of the two asynchronous methods adjusts to the actual level of asynchrony rather than being constrained by the worst-case. Numerical experiments demonstrate a strong practical performance of our asynchronous algorithms.
Paper Structure (34 sections, 14 theorems, 134 equations, 11 figures, 2 algorithms)

This paper contains 34 sections, 14 theorems, 134 equations, 11 figures, 2 algorithms.

Table of Contents

  1. Introduction
  2. Problem Formulation and Prox-DGD
  3. Consensus Optimization
  4. Synchronous Prox-DGD
  5. Optimality of the Fixed Points of Prox-DGD
  6. Asynchronous Prox-DGD
  7. Algorithm Description
  8. Convergence Analysis
  9. Comparison with related methods
  10. Convergence in terms of the Realized Level of Asynchrony
  11. Asynchronous DGD-ATC for Smooth Problems
  12. Algorithm Description
  13. Convergence Analysis
  14. Numerical Experiments
  15. Conclusion
  16. ...and 19 more sections

Key Result

Lemma 1

Suppose that Assumptions asm:optsolexist--asm:prob hold. For any fixed point $\mathbf{x}^\star$ of Prox-DGD, it holds that $F(\mathbf{x}^\star)\le F_{\operatorname{opt}}$ and where $\bar{x}^\star=\frac{1}{n}\sum_{i\in\mathcal{V}} x_i^\star$ and $\beta=\max\{|\lambda_2(W)|, |\lambda_n(W)|\}\in [0,1)$.

Figures (11)

  • Figure 1: distribution of real delays (30 nodes).
  • Figure 2: realized $m^k$ under different delay patterns
  • Figure 3: Smooth problems ($\lambda_1=0$): comparison among asynchronous methods.
  • Figure 4: Smooth problem ($\lambda_1=0$): comparison among asynchronous and synchronous Prox-DGD and DGD-ATC.
  • Figure 5: Non-smooth problem ($\lambda_1=0$): comparison among asynchronous methods.
  • ...and 6 more figures

Theorems & Definitions (27)

  • Definition 1: averaging matrix
  • Lemma 1
  • proof
  • Lemma 2
  • proof
  • Theorem 1: total asynchrony
  • proof
  • Theorem 2: partial asynchrony
  • proof
  • Remark 1: effect of network topology
  • ...and 17 more