Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Accelerated Alternating Direction Method of Multipliers Gradient Tracking for Distributed Optimization

Eduardo Sebastián, Mauro Franceschelli, Andrea Gasparri, Eduardo Montijano, Carlos Sagüés

TL;DR

This work tackles distributed consensus optimization over static undirected networks by proposing an accelerated algorithm that fuses momentum with ADMM and gradient tracking (A2DMM-GT). By modeling the dynamics as an interconnection of two singularly perturbed systems, the authors show that momentum on the average dynamics and on the auxiliary variables yields faster convergence than the non-accelerated ADMM-GT, while retaining robustness and low per-iteration cost. They establish global exponential stability under standard assumptions and provide a convergence-rate argument supported by simulations on quadratic and logistic-regression tasks, where A2DMM-GT outperforms existing first-order distributed protocols given the same computational and communication budget. The approach offers a practical, robust route to faster distributed optimization, with future work on step-size adaptation, time-varying directed graphs, and constrained settings.

Abstract

This paper presents a novel accelerated distributed algorithm for unconstrained consensus optimization over static undirected networks. The proposed algorithm combines the benefits of acceleration from momentum, the robustness of the alternating direction method of multipliers, and the computational efficiency of gradient tracking to surpass existing state-of-the-art methods in convergence speed, while preserving their computational and communication cost. First, we prove that, by applying momentum on the average dynamic consensus protocol over the estimates and gradient, we can study the algorithm as an interconnection of two singularly perturbed systems: the outer system connects the consensus variables and the optimization variables, and the inner system connects the estimates of the optimum and the auxiliary optimization variables. Next, we prove that, by adding momentum to the auxiliary dynamics, our algorithm always achieves faster convergence than the achievable linear convergence rate for the non-accelerated alternating direction method of multipliers gradient tracking algorithm case. Through simulations, we numerically show that our accelerated algorithm surpasses the existing accelerated and non-accelerated distributed consensus first-order optimization protocols in convergence speed.

Accelerated Alternating Direction Method of Multipliers Gradient Tracking for Distributed Optimization

TL;DR

This work tackles distributed consensus optimization over static undirected networks by proposing an accelerated algorithm that fuses momentum with ADMM and gradient tracking (A2DMM-GT). By modeling the dynamics as an interconnection of two singularly perturbed systems, the authors show that momentum on the average dynamics and on the auxiliary variables yields faster convergence than the non-accelerated ADMM-GT, while retaining robustness and low per-iteration cost. They establish global exponential stability under standard assumptions and provide a convergence-rate argument supported by simulations on quadratic and logistic-regression tasks, where A2DMM-GT outperforms existing first-order distributed protocols given the same computational and communication budget. The approach offers a practical, robust route to faster distributed optimization, with future work on step-size adaptation, time-varying directed graphs, and constrained settings.

Abstract

This paper presents a novel accelerated distributed algorithm for unconstrained consensus optimization over static undirected networks. The proposed algorithm combines the benefits of acceleration from momentum, the robustness of the alternating direction method of multipliers, and the computational efficiency of gradient tracking to surpass existing state-of-the-art methods in convergence speed, while preserving their computational and communication cost. First, we prove that, by applying momentum on the average dynamic consensus protocol over the estimates and gradient, we can study the algorithm as an interconnection of two singularly perturbed systems: the outer system connects the consensus variables and the optimization variables, and the inner system connects the estimates of the optimum and the auxiliary optimization variables. Next, we prove that, by adding momentum to the auxiliary dynamics, our algorithm always achieves faster convergence than the achievable linear convergence rate for the non-accelerated alternating direction method of multipliers gradient tracking algorithm case. Through simulations, we numerically show that our accelerated algorithm surpasses the existing accelerated and non-accelerated distributed consensus first-order optimization protocols in convergence speed.
Paper Structure (8 sections, 5 theorems, 26 equations, 2 figures, 2 algorithms)

This paper contains 8 sections, 5 theorems, 26 equations, 2 figures, 2 algorithms.

Table of Contents

  1. Introduction
  2. Preliminaries
  3. Problem Formulation
  4. Singularly Perturbed Systems
  5. ADMM Gradient Tracking
  6. Accelerated ADMM Gradient Tracking
  7. Illustrative example
  8. Conclusions

Key Result

Theorem 1

Let with $\color{black}{\color{black}{\mathbf{p}}\color{black}\space}\color{black}\space \in \mathbb{R}^n$, $\color{black}{\mathbf{q}}\color{black}\space \in \mathbb{R}^m$, $\gamma > 0$ and $f(\bullet), g(\bullet)$ Lipschitz continuous uniformly over $t$. Let assume that there exists a function $\color{ We call reduced system the system given by and boundary system layer with $\color{black}{\til

Figures (2)

  • Figure 1: Representation of Algorithm \ref{['al:accelerated']} as an interconnection of two singularly perturbed systems. The outer singularly perturbed system connects the dynamics of the consensus variables $\mathbf{r}^t$ and the estimates/auxiliary variables $\mathbf{w}^t$. The inner singularly perturbed system connects the dynamics of the estimates $\mathbf{x}^t$ and the auxiliary variables $\mathbf{z}^t$.
  • Figure 2: Evolution of the error with time for the five protocols under comparison. Left is the quadratic programming setup, right is the linear classification setup.

Theorems & Definitions (9)

  • Theorem 1: Theorem II.3, carnevale2023admm
  • Lemma 1
  • proof
  • Lemma 2
  • proof
  • Proposition 1
  • proof
  • Theorem 2
  • proof