Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Inexact Augmented Lagrangian Methods for Conic Programs: Quadratic Growth and Linear Convergence

Feng-Yi Liao, Lijun Ding, Yang Zheng

TL;DR

Both primal and dual iterates of the ALMs converge linearly contingent solely upon the assumption of strict complementarity and a bounded solution set, providing a positive answer to an open question regarding the asymptotically linear convergence of the primal iterates of ALMs applied to semidefinite optimization.

Abstract

Augmented Lagrangian Methods (ALMs) are widely employed in solving constrained optimizations, and some efficient solvers are developed based on this framework. Under the quadratic growth assumption, it is known that the dual iterates and the Karush-Kuhn-Tucker (KKT) residuals of ALMs applied to semidefinite programs (SDPs) converge linearly. In contrast, the convergence rate of the primal iterates has remained elusive. In this paper, we resolve this challenge by establishing new $\textit{quadratic growth}$ and $\textit{error bound}$ properties for primal and dual SDPs under the strict complementarity condition. Our main results reveal that both primal and dual iterates of the ALMs converge linearly contingent solely upon the assumption of strict complementarity and a bounded solution set. This finding provides a positive answer to an open question regarding the asymptotically linear convergence of the primal iterates of ALMs applied to semidefinite optimization.

Inexact Augmented Lagrangian Methods for Conic Programs: Quadratic Growth and Linear Convergence

TL;DR

Both primal and dual iterates of the ALMs converge linearly contingent solely upon the assumption of strict complementarity and a bounded solution set, providing a positive answer to an open question regarding the asymptotically linear convergence of the primal iterates of ALMs applied to semidefinite optimization.

Abstract

Augmented Lagrangian Methods (ALMs) are widely employed in solving constrained optimizations, and some efficient solvers are developed based on this framework. Under the quadratic growth assumption, it is known that the dual iterates and the Karush-Kuhn-Tucker (KKT) residuals of ALMs applied to semidefinite programs (SDPs) converge linearly. In contrast, the convergence rate of the primal iterates has remained elusive. In this paper, we resolve this challenge by establishing new and properties for primal and dual SDPs under the strict complementarity condition. Our main results reveal that both primal and dual iterates of the ALMs converge linearly contingent solely upon the assumption of strict complementarity and a bounded solution set. This finding provides a positive answer to an open question regarding the asymptotically linear convergence of the primal iterates of ALMs applied to semidefinite optimization.

Paper Structure

This paper contains 32 sections, 23 theorems, 116 equations, 5 figures.

Table of Contents

  1. Introduction
  2. Preliminaries
  3. Strong duality and strict complementarity
  4. Proximal point methods (PPMs)
  5. Quadratic Growth in SDPs
  6. Growth and error bound properties
  7. Proof sketches for \ref{['theorem:growth-error-bound-dual', 'theorem:growth-error-bound']}
  8. Augmented Lagrangian Methods (ALMs) for SDPs
  9. Inexact Augmented Lagrangian Method
  10. Linear convergences of primal and dual iterates in ALM
  11. Numerical experiments
  12. Conclusion
  13. Optimization Background
  14. Notations
  15. Standard notion of strict complementarity in SDPs
  16. ...and 17 more sections

Key Result

Lemma 1

Let $S = \mathop{\mathrm{\arg\!\min}}\limits_{x\in \mathbb{R}^n} \, f(x)$ and suppose $S \neq \emptyset$. Let $\{x_k\}_{k\geq 0}$ be a sequence from eq:iPPM.

Figures (5)

  • Figure 1: Numerical convergence behavior of inexact ALM \ref{['eq:alm-update-conic-primal-both']} for Max-Cut and linear SVM. The symbol $\epsilon_3$ denotes the KKT residuals $\epsilon_3 = \max\{\eta_1,\eta_2,\eta_3,\eta_4,\eta_5\}$.
  • Figure 2: The quadratric growth property of the exact penalty function $f(y)=-b^\mathsf{ T} y + \rho \max \{0,\lambda_{\max}(\mathcal{A}^*(y) - C)\}$ where $\rho = 4$ and $f^\star = 0$. The optimal solution set $S = \{(0,0)\}.$ In \ref{['fig:QG-3D']}, the yellow region represents the linear part where only $-b^\mathsf{ T} y$ is active, resulting $\rho \max \{0,\lambda_{\max}(\mathcal{A}^*(y) - C)\} = 0$, the blue region encompasses both the linear and the nonlinear parts, and the green surf is the square of the distance to the optimal solution set with $\kappa = 0.3$. \ref{['fig:QG-Sectional']} shows the sectional view of $f$ along the direction $y_1y_2 - y_1 - y_2 = 0$, which can be characterized as the rational function $f(y_1) = \frac{-y_1^2}{y_1-1}$.
  • Figure 3: Nuremical experiments for Max-Cut with different instances and various augmented Lagrangian parameter $r > 0.$
  • Figure 4: Nuremical experiments for linear SVM with different instances and various augmented Lagrangian parameter $r > 0.$
  • Figure 5: Nuremical experiments for Lasso with different instances and various augmented Lagrangian parameter $r > 0.$

Theorems & Definitions (43)

  • Definition 1: Strong duality
  • Definition 2: Dual strict complementarity for SDPs
  • Lemma 1
  • Lemma 2
  • Theorem 1: Growth properties in the primal
  • Theorem 2: Growth properties in the dual
  • Remark 1
  • Remark 2: Conic programs
  • Lemma 3
  • Remark 3
  • ...and 33 more