Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

An accelerated proximal PRS-SQP algorithm with dual ascent-descent procedures for smooth composite optimization

Jiachen Jin, Guodong Ma, Jinbao Jian

TL;DR

This work addresses smooth composite optimization of the form $\min_x f(x) + g(Ax)$ by introducing the HAP-PRS-SQP algorithm, which unifies augmented Lagrangian dual ascent and dual descent updates into a single framework with flexible stepsizes $r$ and $s$ where $r+s \neq 0$. A hybrid acceleration that blends inertial extrapolation and back substitution is embedded between algorithmic steps to enhance convergence. The authors prove convergence under KL assumptions and provide rates depending on the KL exponent, demonstrating that the dual update direction does not compromise convergence. Numerical experiments on regularized binary classification and smooth LASSO validate the method, showing improved convergence speed and stability over existing splitting SQP approaches and gradient methods, with robust performance across different dual-update regimes.

Abstract

Conventional wisdom in composite optimization suggests augmented Lagrangian dual ascent (ALDA) in Peaceman-Rachford splitting (PRS) methods for dual feasibility. However, ALDA may fail when the primal iterate is a local minimum, a stationary point, or a coordinatewise solution of the highly nonconvex augmented Lagrangian function. Splitting sequential quadratic programming (SQP) methods utilize augmented Lagrangian dual descent (ALDD) to directly minimize the primal residual, circumventing the limitations of ALDA and achieving faster convergence in smooth optimization. This paper aims to present a fairly accessible generalization of two contrasting dual updates, ALDA and ALDD, for smooth composite optimization. A key feature of our PRS-SQP algorithm is its dual ascent-descent procedure, which provides a free direction rule for the dual updates and a new insight to explain the counterintuitive convergence behavior. Furthermore, we incorporate a hybrid acceleration technique that combines inertial extrapolation and back substitution to improve convergence. Theoretically, we establish the feasibility for a wider range of acceleration factors than previously known and derive convergence rates within the Kurdyka- Lojasiewicz framework. Numerical experiments validate the effectiveness and stability of the proposed method in various dual-update scenarios.

An accelerated proximal PRS-SQP algorithm with dual ascent-descent procedures for smooth composite optimization

TL;DR

This work addresses smooth composite optimization of the form by introducing the HAP-PRS-SQP algorithm, which unifies augmented Lagrangian dual ascent and dual descent updates into a single framework with flexible stepsizes and where . A hybrid acceleration that blends inertial extrapolation and back substitution is embedded between algorithmic steps to enhance convergence. The authors prove convergence under KL assumptions and provide rates depending on the KL exponent, demonstrating that the dual update direction does not compromise convergence. Numerical experiments on regularized binary classification and smooth LASSO validate the method, showing improved convergence speed and stability over existing splitting SQP approaches and gradient methods, with robust performance across different dual-update regimes.

Abstract

Conventional wisdom in composite optimization suggests augmented Lagrangian dual ascent (ALDA) in Peaceman-Rachford splitting (PRS) methods for dual feasibility. However, ALDA may fail when the primal iterate is a local minimum, a stationary point, or a coordinatewise solution of the highly nonconvex augmented Lagrangian function. Splitting sequential quadratic programming (SQP) methods utilize augmented Lagrangian dual descent (ALDD) to directly minimize the primal residual, circumventing the limitations of ALDA and achieving faster convergence in smooth optimization. This paper aims to present a fairly accessible generalization of two contrasting dual updates, ALDA and ALDD, for smooth composite optimization. A key feature of our PRS-SQP algorithm is its dual ascent-descent procedure, which provides a free direction rule for the dual updates and a new insight to explain the counterintuitive convergence behavior. Furthermore, we incorporate a hybrid acceleration technique that combines inertial extrapolation and back substitution to improve convergence. Theoretically, we establish the feasibility for a wider range of acceleration factors than previously known and derive convergence rates within the Kurdyka- Lojasiewicz framework. Numerical experiments validate the effectiveness and stability of the proposed method in various dual-update scenarios.
Paper Structure (11 sections, 10 theorems, 113 equations, 3 figures, 2 tables, 1 algorithm)

This paper contains 11 sections, 10 theorems, 113 equations, 3 figures, 2 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Peaceman-Rachford splitting methods
  3. Augmented Lagrangian dual descent methods
  4. Contributions
  5. Preliminaries
  6. Algorithm design
  7. Convergence analysis
  8. Numerical experiments
  9. Regularized binary classification problem
  10. Smooth LASSO
  11. Conclusions

Key Result

Lemma 1

Nesterov2018 For a continuous differentiable function $h: \mathbb{R}^n\to\mathbb{R}$, its gradient is Lipschitz continuous with the modulus $L>0$, then,

Figures (3)

  • Figure 1: Performance profiles of HAP-PRS-SQP with different $\alpha$ on Problem \ref{['L2reg']} with $n=T=100$.
  • Figure 2: Changes of the objective function value and the relative accuracy for Problem \ref{['L2reg']} with $n=T=500$.
  • Figure 3: Numerical results on the Huber smooth LASSO problem

Theorems & Definitions (22)

  • Lemma 1
  • Definition 1
  • Lemma 2
  • Definition 2
  • Remark 1
  • Remark 2
  • Lemma 3
  • proof
  • Lemma 4
  • proof
  • ...and 12 more