Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Stability of Primal-Dual Gradient Flow Dynamics for Multi-Block Convex Optimization Problems

Ibrahim K. Ozaslan, Panagiotis Patrinos, Mihailo R. Jovanović

TL;DR

The paper develops primal-dual gradient flow dynamics built on a proximal augmented Lagrangian to solve multi-block convex optimization with generalized consensus constraints, including nondifferentiable terms. It establishes GAS under feasibility/convexity, LES under PL and polyhedral/group-sparse nonsmooth terms, and GES under a range-space condition combined with strong primal convexity via the augmented Lagrangian, using novel Lyapunov constructions V1, V2, and their sum V3. The analysis leverages LaSalle's invariance principle, Hoffman-type error bounds, and a differentiable dual gradient to obtain global exponential rates without requiring strong convexity of the entire objective; it also demonstrates the necessity of the Lipschitz assumption for GES. The framework supports distributed implementations (via network Laplacians and neighbor communications) and yields exponential convergence in broad settings, with several illustrative examples including decentralized lasso, robust PCA, covariance completion, and sparse group lasso. Overall, the work provides a systematic, broadly applicable alternative to ADMM for large-scale multi-block problems, offering explicit exponential convergence guarantees under relatively mild structural conditions and enabling parallel/distributed computation.

Abstract

We examine stability properties of primal-dual gradient flow dynamics for composite convex optimization problems with multiple, possibly nonsmooth, terms in the objective function under the generalized consensus constraint. The proposed dynamics are based on the proximal augmented Lagrangian and they provide a viable alternative to ADMM which faces significant challenges from both analysis and implementation viewpoints in large-scale multi-block scenarios. In contrast to customized algorithms with individualized convergence guarantees, we develop a systematic approach for solving a broad class of challenging composite optimization problems. We leverage various structural properties to establish global (exponential) convergence guarantees for the proposed dynamics. Our assumptions are much weaker than those required to prove (exponential) stability of primal-dual dynamics as well as (linear) convergence of discrete-time methods such as standard two-block and multi-block ADMM and EXTRA algorithms. Finally, we show necessity of some of our structural assumptions for exponential stability and provide computational experiments to demonstrate the convenience of the proposed approach for parallel and distributed computing applications.

Stability of Primal-Dual Gradient Flow Dynamics for Multi-Block Convex Optimization Problems

TL;DR

The paper develops primal-dual gradient flow dynamics built on a proximal augmented Lagrangian to solve multi-block convex optimization with generalized consensus constraints, including nondifferentiable terms. It establishes GAS under feasibility/convexity, LES under PL and polyhedral/group-sparse nonsmooth terms, and GES under a range-space condition combined with strong primal convexity via the augmented Lagrangian, using novel Lyapunov constructions V1, V2, and their sum V3. The analysis leverages LaSalle's invariance principle, Hoffman-type error bounds, and a differentiable dual gradient to obtain global exponential rates without requiring strong convexity of the entire objective; it also demonstrates the necessity of the Lipschitz assumption for GES. The framework supports distributed implementations (via network Laplacians and neighbor communications) and yields exponential convergence in broad settings, with several illustrative examples including decentralized lasso, robust PCA, covariance completion, and sparse group lasso. Overall, the work provides a systematic, broadly applicable alternative to ADMM for large-scale multi-block problems, offering explicit exponential convergence guarantees under relatively mild structural conditions and enabling parallel/distributed computation.

Abstract

We examine stability properties of primal-dual gradient flow dynamics for composite convex optimization problems with multiple, possibly nonsmooth, terms in the objective function under the generalized consensus constraint. The proposed dynamics are based on the proximal augmented Lagrangian and they provide a viable alternative to ADMM which faces significant challenges from both analysis and implementation viewpoints in large-scale multi-block scenarios. In contrast to customized algorithms with individualized convergence guarantees, we develop a systematic approach for solving a broad class of challenging composite optimization problems. We leverage various structural properties to establish global (exponential) convergence guarantees for the proposed dynamics. Our assumptions are much weaker than those required to prove (exponential) stability of primal-dual dynamics as well as (linear) convergence of discrete-time methods such as standard two-block and multi-block ADMM and EXTRA algorithms. Finally, we show necessity of some of our structural assumptions for exponential stability and provide computational experiments to demonstrate the convenience of the proposed approach for parallel and distributed computing applications.
Paper Structure (32 sections, 13 theorems, 136 equations, 4 figures)

This paper contains 32 sections, 13 theorems, 136 equations, 4 figures.

Table of Contents

  1. Introduction
  2. Background and motivation
  3. Lagrange saddle function
  4. Proximal augmented Lagrangian
  5. Primal-dual gradient flow dynamics
  6. Distributed implementation
  7. Main results
  8. Related works and discussion
  9. Primal-dual gradient flow dynamics
  10. ADMM
  11. Comparison of related works on a consensus optimization problem
  12. Proof of main results
  13. Proof of Theorem \ref{['theorem.gas']}: Global asymptotic stability
  14. Proof of Theorem \ref{['theorem.sges']}: Local exponential stability
  15. Lagrange dual problem
  16. ...and 17 more sections

Key Result

Theorem 1

Let Assumption ass.zero hold. The set of equilibrium points $\Psi^\star$ of PD gradient flow dynamics eq.dyn, characterized by KKT conditions eq.kkt, is globally asymptotically stable and the solution to eq.dyn converges to a point in this set.

Figures (4)

  • Figure 1: Topology of the underlying communication network in distributed lasso problem \ref{['ex.dist_opt']} and the Semi-GES of the distributed dynamics \ref{['eq.decentralized_dyn']}. $F(t)$ denotes the objective value of \ref{['ex.dist_opt.equ']} at time $t$. The reference solution is obtained by using CVX.
  • Figure 2: Semi-GES of dynamics \ref{['eq.dyn_ex2']} for principle component pursuit problem \ref{['ex.pcp']}. $F(t)$ denotes the objective value of \ref{['ex.pcp']} at time $t$. The reference solution is obtained by performing $10^4$ iterations of VASALM algorithm taoyua11.
  • Figure 3: Semi-GES of dynamics \ref{['eq.dyn_ex3']} for covariance completion problem \ref{['ex.cc']}. $F(t)$ denotes the objective value of \ref{['ex.cc']} at time $t$. The reference solution is obtained by using CVX.
  • Figure 4: Semi-GES of dynamics \ref{['eq.dyn_ex4']} for sparse group lasso problem \ref{['ex.sgl']}. $F(t)$ denotes the objective value of \ref{['ex.sgl']} at time $t$. The reference solutions are obtained by using CVX.

Theorems & Definitions (41)

  • Theorem 1: GAS
  • proof
  • Remark 1
  • Remark 2
  • Theorem 2: LES
  • proof
  • Remark 3
  • Corollary 1: Semi-GES
  • proof
  • Theorem 3: GES
  • ...and 31 more