Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

The Augmented Lagrangian Methods: Overview and Recent Advances

Kangkang Deng, Rui Wang, Zhenyuan Zhu, Junyu Zhang, Zaiwen Wen

TL;DR

This survey provides a unified view of augmented Lagrangian methods (ALM), tracing the construction of AL functions from the Hestenes–Powell–Rockafellar framework and detailing strong duality properties that underpin dual-ascent interpretations of ALM. It covers convex and nonconvex, smooth and nonsmooth, as well as composite and discrete problems, introducing variants such as linearized, proximal, accelerated, stochastic, and manifold ALMs, along with ADMM and primal–dual schemes. The paper surveys convergence guarantees (global and local) and iteration-complexity results, and demonstrates practical subproblem solvers (proximal gradient, semismooth Newton) through wide-ranging applications including NLP, SDP, sparse optimization, RL, and distributed optimization. It also discusses exact penalty representations, multiplier stability, and the role of the Moreau envelope in unifying AL constructions. The work serves as both a theoretical reference and a practical guide for implementing ALM in large-scale, real-world constrained optimization problems, highlighting current challenges and promising future directions in single-loop, stochastic, and structure-exploiting variants.

Abstract

Large-scale constrained optimization is pivotal in modern scientific, engineering, and industrial computation, often involving complex systems with numerous variables and constraints. This paper provides a unified and comprehensive perspective on constructing augmented Lagrangian functions (based on Hestenes-Powell-Rockafellar augmented Lagrangian) for various optimization problems, including nonlinear programming and convex and nonconvex composite programming. We present the augmented Lagrangian method (ALM), covering its theoretical foundations in both convex and nonconvex cases, and discuss several successful examples and applications. Recent advancements have extended ALM's capabilities to handle nonconvex constraints and ensure global convergence to first and second-order stationary points. For nonsmooth convex problems, ALM utilizes proximal operations, preserving desirable properties such as locally linear convergence rates. Furthermore, recent progress has refined the complexity analysis for ALM and tackled challenging integer programming instances. This review aims to offer a thorough understanding of ALM's benefits and limitations, exploring different ALM variants designed to enhance convergence and computational performance. We also illustrate effective algorithms for ALM subproblems across different types of optimization problems and highlight practical implementations in several fields.

The Augmented Lagrangian Methods: Overview and Recent Advances

TL;DR

This survey provides a unified view of augmented Lagrangian methods (ALM), tracing the construction of AL functions from the Hestenes–Powell–Rockafellar framework and detailing strong duality properties that underpin dual-ascent interpretations of ALM. It covers convex and nonconvex, smooth and nonsmooth, as well as composite and discrete problems, introducing variants such as linearized, proximal, accelerated, stochastic, and manifold ALMs, along with ADMM and primal–dual schemes. The paper surveys convergence guarantees (global and local) and iteration-complexity results, and demonstrates practical subproblem solvers (proximal gradient, semismooth Newton) through wide-ranging applications including NLP, SDP, sparse optimization, RL, and distributed optimization. It also discusses exact penalty representations, multiplier stability, and the role of the Moreau envelope in unifying AL constructions. The work serves as both a theoretical reference and a practical guide for implementing ALM in large-scale, real-world constrained optimization problems, highlighting current challenges and promising future directions in single-loop, stochastic, and structure-exploiting variants.

Abstract

Large-scale constrained optimization is pivotal in modern scientific, engineering, and industrial computation, often involving complex systems with numerous variables and constraints. This paper provides a unified and comprehensive perspective on constructing augmented Lagrangian functions (based on Hestenes-Powell-Rockafellar augmented Lagrangian) for various optimization problems, including nonlinear programming and convex and nonconvex composite programming. We present the augmented Lagrangian method (ALM), covering its theoretical foundations in both convex and nonconvex cases, and discuss several successful examples and applications. Recent advancements have extended ALM's capabilities to handle nonconvex constraints and ensure global convergence to first and second-order stationary points. For nonsmooth convex problems, ALM utilizes proximal operations, preserving desirable properties such as locally linear convergence rates. Furthermore, recent progress has refined the complexity analysis for ALM and tackled challenging integer programming instances. This review aims to offer a thorough understanding of ALM's benefits and limitations, exploring different ALM variants designed to enhance convergence and computational performance. We also illustrate effective algorithms for ALM subproblems across different types of optimization problems and highlight practical implementations in several fields.
Paper Structure (88 sections, 21 theorems, 430 equations, 7 figures, 5 tables, 2 algorithms)

This paper contains 88 sections, 21 theorems, 430 equations, 7 figures, 5 tables, 2 algorithms.

Table of Contents

  1. AMS subject classifications.
  2. Introduction
  3. Notation
  4. Organization
  5. Augmented Lagrangian function and strong duality
  6. General form constrained optimization problem
  7. KKT condition and Lagrangian duality
  8. Classic augmented Lagrangian function
  9. Alternative AL construction via shift penalty
  10. Alternative AL construction via general augmenting function
  11. Other AL function
  12. Saddle point formulation
  13. Exact penalty representation
  14. Convex composite optimization problem
  15. Preliminaries
  16. ...and 73 more sections

Key Result

Theorem 2.6

Suppose $f(x)$ and $c_i(x), i\in\mathcal{E}\cup\mathcal{I}$ are continuously differentiable and let $x^*$ be a local minimum of problem prob. If LICQ or MFCQ is satisfied at $x^*$, then there exist Lagrange multipliers $\lambda_i^*, i \in \mathcal{E}$ and $\mu_i^*\geq0,i\in\mathcal{I}$ such that

Figures (7)

  • Figure 1: Comparison of the performance of different algorithms on 20 problems in the form of \ref{['sec8:eq:nlp']}
  • Figure 2: Comparisons among fully primal-dual methods on basis pursuit problems.
  • Figure 3: The performance profiles of SDPDAL and SDPNAL+ for Max-Cut problems (G01--G54)
  • Figure 4: Phantom: the PSNR of FBP and ALPDSN are $19.26$, $35.37$, respectively.
  • Figure 5: Comparison between ALPDSN and other first-order methods on Phantom.
  • ...and 2 more figures

Theorems & Definitions (54)

  • Definition 2.1: active set
  • Definition 2.2: LICQ bertsekas1997nonlinear
  • Definition 2.3: MFCQ mangasarian1967fritz
  • Definition 2.4: Slater's condition boyd2004convex
  • Definition 2.5: Limiting Fréchet subdifferential Kruger2003
  • Theorem 2.6
  • Theorem 2.7
  • Theorem 2.8
  • Definition 2.9
  • Theorem 2.10
  • ...and 44 more