Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

An Accelerated Primal Dual Algorithm with Backtracking for Decentralized Constrained Optimization

Qiushui Xu, Necdet Serhat Aybat, Mert Gürbüzbalaban

TL;DR

D-APDB is the first distributed method with backtracking that achieves the optimal convergence rate for the class of constrained composite convex optimization problems and is established as the first distributed method with backtracking that achieves the optimal convergence rate for the class of constrained composite convex optimization problems.

Abstract

We propose a distributed accelerated primal-dual method with backtracking (D-APDB) for cooperative multi-agent constrained consensus optimization problems over an undirected network of agents, where only those agents connected by an edge can directly communicate to exchange large-volume data vectors using a high-speed, short-range communication protocol, e.g., WiFi, and we also assume that the network allows for one-hop simple information exchange beyond immediate neighbors as in LoRaWAN protocol. The objective is to minimize the sum of agent-specific composite convex functions over agent-specific private constraint sets. Unlike existing decentralized primal-dual methods that require knowledge of the Lipschitz constants, D-APDB automatically adapts to unknown smoothness constants by employing a distributed backtracking step-size search. Each agent relies only on first-order oracles associated with its own objective and constraint functions and on local communications with the neighboring agents, without any prior knowledge of Lipschitz constants. We establish $\mathcal{O}(1/K)$ convergence guarantees for sub-optimality, infeasibility and consensus violation, under standard assumptions on smoothness and on the connectivity of the communication graph. To our knowledge, when nodes have private constraints, especially when they are nonlinear convex constraints onto which projections are not cheap to compute, D-APDB is the first distributed method with backtracking that achieves the optimal convergence rate for the class of constrained composite convex optimization problems. We also provide numerical results for D-APDB on a distributed QCQP problem and distributed primal SVM training to illustrate the potential performance gains that can be achieved by D-APDB.

An Accelerated Primal Dual Algorithm with Backtracking for Decentralized Constrained Optimization

TL;DR

D-APDB is the first distributed method with backtracking that achieves the optimal convergence rate for the class of constrained composite convex optimization problems and is established as the first distributed method with backtracking that achieves the optimal convergence rate for the class of constrained composite convex optimization problems.

Abstract

We propose a distributed accelerated primal-dual method with backtracking (D-APDB) for cooperative multi-agent constrained consensus optimization problems over an undirected network of agents, where only those agents connected by an edge can directly communicate to exchange large-volume data vectors using a high-speed, short-range communication protocol, e.g., WiFi, and we also assume that the network allows for one-hop simple information exchange beyond immediate neighbors as in LoRaWAN protocol. The objective is to minimize the sum of agent-specific composite convex functions over agent-specific private constraint sets. Unlike existing decentralized primal-dual methods that require knowledge of the Lipschitz constants, D-APDB automatically adapts to unknown smoothness constants by employing a distributed backtracking step-size search. Each agent relies only on first-order oracles associated with its own objective and constraint functions and on local communications with the neighboring agents, without any prior knowledge of Lipschitz constants. We establish convergence guarantees for sub-optimality, infeasibility and consensus violation, under standard assumptions on smoothness and on the connectivity of the communication graph. To our knowledge, when nodes have private constraints, especially when they are nonlinear convex constraints onto which projections are not cheap to compute, D-APDB is the first distributed method with backtracking that achieves the optimal convergence rate for the class of constrained composite convex optimization problems. We also provide numerical results for D-APDB on a distributed QCQP problem and distributed primal SVM training to illustrate the potential performance gains that can be achieved by D-APDB.

Paper Structure

This paper contains 17 sections, 16 theorems, 105 equations, 3 figures, 2 algorithms.

Table of Contents

  1. Introduction
  2. Notation
  3. Assumptions and the Main Results
  4. Assumptions
  5. Main Results
  6. Motivation and Design of the Proposed Algorithms: D-APDB and D-APDB0
  7. Preliminaries
  8. Distributed implementation of the abstract method
  9. From the abstract method towards D-APDB in Algorithm \ref{['alg:apdb']}
  10. Preliminary Technical Results
  11. Parameter Choices
  12. Convergence guarantees for D-APDB and the Proof of Main Results
  13. Experiments
  14. QCQP problems with $\ell_1$-norm regularization.
  15. Unconstrained $\ell_1$-norm-regularized quadratic programming
  16. ...and 2 more sections

Key Result

Theorem 2.6

Suppose that assmp:fassmp:gassmp:Nassmp:bounded_dual_domain hold, and $\delta,c_\alpha,c_\beta,c_\varsigma>0$ are given such that $\delta+c<1$, where $c\triangleq c_\alpha+c_\beta+c_\varsigma$. Let $(x^*,\theta^*)\in\mathbb{R}^n\times\mathcal{K}^*$ denote an arbitrary primal-dual optimal pair satisf where $(\bar{x}_i^K,\bar{\theta}_i^K)=\sum_{k=0}^{K-1}t_k(x_i^k,\theta_i^k)/\sum_{k=0}^{K-1}t_k$ fo

Figures (3)

  • Figure 1: Comparison of D-APDB against D-APD for solving \ref{['eq:qcqp']} over 20 runs. “Average number of gradient calls per node” means $\frac{1}{N}\sum_{i=1}^N n_i^k$ where $n_i^k$ counts how many calls for gradients for node $i$ at iteration $k$.
  • Figure 2: Comparison of D-APDB0 against D-APD and global_DATOS for solving \ref{['eq:qp']} with 20 simulation.
  • Figure 3: Comparison of D-APDB against D-APD for solving the linear SVM problem in \ref{['eq:svm']} over 20 random problem instances. “Average number of gradient calls per node” means $\frac{1}{N}\sum_{i=1}^N n_i^k$ where $n_i^k$ counts how many calls for gradients for node $i$ at iteration $k$.

Theorems & Definitions (39)

  • Definition 2.1
  • Remark 2.2
  • Definition 2.3
  • Definition 2.4
  • Remark 2.5
  • Theorem 2.6
  • proof
  • Theorem 2.7
  • proof
  • Remark 4.1
  • ...and 29 more