Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Augmented Lagrangian methods for fully convex composite optimization

Alberto De Marchi, Tim Hoheisel, Patrick Mehlitz

TL;DR

This work extends augmented Lagrangian methods to fully convex composite optimization by linking ALM updates to inexact proximal point iterations on the dual problem. It analyzes classical ALM, safeguarded ALMs, and introduces elastic safeguarding, which enlarges the safeguarding set while ensuring $\mu_k\hat{y}^k \to 0$, thereby achieving dual convergence to a Lagrange multiplier when it exists and preserving primal convergence. Key results include global primal convergence to minimizers, convergence of dual iterates to multipliers in the regular case, and refined convergence guarantees for safeguarded schemes via backward-backward splitting interpretations. The approach reconciles the robustness of safeguarding with the optimality properties of classical ALM, offering practical schemes with strong theoretical guarantees for convex composite problems and suggesting extensions to broader nonconvex or generalized settings.

Abstract

This paper is concerned with augmented Lagrangian methods for the treatment of fully convex composite optimization problems. We extend the classical relationship between augmented Lagrangian methods and the proximal point algorithm to the inexact and safeguarded scheme in order to state global primal-dual convergence results. Our analysis distinguishes the regular case, where a stationary minimizer exists, and the irregular case, where all minimizers are nonstationary. Furthermore, we suggest an elastic modification of the standard safeguarding scheme which preserves primal convergence properties while guaranteeing convergence of the dual sequence to a multiplier in the regular situation. Although important for nonconvex problems, the standard safeguarding mechanism leads to weaker convergence guarantees for convex problems than the classical augmented Lagrangian method. Our elastic safeguarding scheme combines the advantages of both while avoiding their shortcomings.

Augmented Lagrangian methods for fully convex composite optimization

TL;DR

This work extends augmented Lagrangian methods to fully convex composite optimization by linking ALM updates to inexact proximal point iterations on the dual problem. It analyzes classical ALM, safeguarded ALMs, and introduces elastic safeguarding, which enlarges the safeguarding set while ensuring , thereby achieving dual convergence to a Lagrange multiplier when it exists and preserving primal convergence. Key results include global primal convergence to minimizers, convergence of dual iterates to multipliers in the regular case, and refined convergence guarantees for safeguarded schemes via backward-backward splitting interpretations. The approach reconciles the robustness of safeguarding with the optimality properties of classical ALM, offering practical schemes with strong theoretical guarantees for convex composite problems and suggesting extensions to broader nonconvex or generalized settings.

Abstract

This paper is concerned with augmented Lagrangian methods for the treatment of fully convex composite optimization problems. We extend the classical relationship between augmented Lagrangian methods and the proximal point algorithm to the inexact and safeguarded scheme in order to state global primal-dual convergence results. Our analysis distinguishes the regular case, where a stationary minimizer exists, and the irregular case, where all minimizers are nonstationary. Furthermore, we suggest an elastic modification of the standard safeguarding scheme which preserves primal convergence properties while guaranteeing convergence of the dual sequence to a multiplier in the regular situation. Although important for nonconvex problems, the standard safeguarding mechanism leads to weaker convergence guarantees for convex problems than the classical augmented Lagrangian method. Our elastic safeguarding scheme combines the advantages of both while avoiding their shortcomings.

Paper Structure

This paper contains 21 sections, 25 theorems, 114 equations, 3 figures, 1 table.

Table of Contents

  1. Introduction
  2. Notation and preliminaries
  3. Fundamentals
  4. Recession cones and functions
  5. On inexact PPA
  6. Lagrangian framework
  7. Fully convex composite optimization
  8. On the value function of a fully convex composite problem
  9. Duality
  10. Classical ALM through the PPA lens
  11. Dual updates as inexact PPA
  12. Convergence analysis
  13. Dual updates as proximal-gradient ascent
  14. Safeguarded ALM through the PPA lens
  15. Dual updates as inexact backward-backward splitting
  16. ...and 6 more sections

Key Result

Lemma 2.1

Let $\{y^k\}\subseteq\mathbb{R}^m$ be a sequence, and let $Y\subseteq\mathbb{R}^m$ be a nonempty set. Assume that each accumulation point of $\{y^k\}$ belongs to $Y$ and that, for each $y\in Y$, $\{\Vert y^k-y \Vert\}$ converges. Then there exists $\bar{y}\in Y$ such that $y^k\to\bar{y}$.

Figures (3)

  • Figure 7.1: Regular example: Convex problem with stationary solution \ref{['problem:cvx_reg']}. All variants converge to the primal-dual solution $(\bar{x},\bar{y})$ except the one with fixed safeguard and fixed penalty parameter, because the selected safeguarding set $Y_\textup{sg}$ is too small (middle and right panels). The elastic variant displays two regimes: first, it behaves as with rigid safeguard when parameter $\rho_k$ is still small, then it switches to a faster convergence as for the classical scheme without safeguard (middle panel). Such transition mitigates the number of penalty parameter updates (left panel).
  • Figure 7.2: Irregular example: Convex problem without stationary solution \ref{['problem:cvx_deg']}. Legend as in \ref{['fig:cvx_reg']}. All variants converge to the solution $\bar{x}$ except the one with fixed penalty parameter and rigid safeguard (middle panel). All variants with adaptive penalty quickly converge to the solution, with $\mu_k\downarrow 0$ (left panel). The classical variant with fixed penalty parameter slowly converges, thanks to $|y^k|\to\infty$ (right panel).
  • Figure 7.3: Kanzow--Steck example: Nonconvex formulation of convex problem with stationary solution \ref{['problem:ks_reg']}. Legend as in \ref{['fig:cvx_reg']}. The nonconvex subproblem is warm-started at $x^{k-1}$ (top panels) or cold-started at $1$ (bottom panels). When warm-started, all variants with adaptive penalty converge to the solution, and only the one with rigid safeguard exhibits $\mu_k\downarrow 0$, while variants with fixed penalty parameter fail to converge, as they generate primal iterates $x^k$ approaching zero from below. When cold-started, the behaviors observed on the regular example are resumed.

Theorems & Definitions (61)

  • Lemma 2.1
  • proof
  • Lemma 2.2
  • Lemma 2.3
  • proof
  • Proposition 2.4
  • proof
  • Lemma 3.2
  • proof
  • Remark 3.3
  • ...and 51 more