Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

A Bundle-based Augmented Lagrangian Framework: Algorithm, Convergence, and Primal-dual Principles

Feng-Yi Liao, Yang Zheng

TL;DR

This work introduces BALA, a single-loop bundle-based augmented Lagrangian method for constrained convex optimization, leveraging a bundle of past iterates to form a tractable inner approximation Ω_k ⊆ Ω and enabling exact subproblem solves. The authors establish sublinear convergence for primal feasibility, primal objective, and dual iterates, and prove linear convergence under quadratic growth and quadratic closeness of the dual approximation, with applicability to conic programs including semidefinite programs. The analysis reveals deep connections between BALA and the proximal bundle method (PBM) as well as inexact ALM, providing a unified primal–dual perspective. Numerical experiments on SDPs demonstrate that BALA outperforms conditional-gradient ALM variants, achieving high accuracy and exhibiting linear convergence in practice, highlighting its practical impact for large-scale conic optimization.

Abstract

We propose a new bundle-based augmented Lagrangian framework for solving constrained convex problems. Unlike the classical (inexact) augmented Lagrangian method (ALM) that has a nested double-loop structure, our framework features a $\textit{single-loop}$ process. Motivated by the proximal bundle method (PBM), we use a $\textit{bundle}$ of past iterates to approximate the subproblem in ALM to get a computationally efficient update at each iteration. We establish sub-linear convergences for primal feasibility, primal cost values, and dual iterates under mild assumptions. With further regularity conditions, such as quadratic growth, our algorithm enjoys $\textit{linear}$ convergences. Importantly, this linear convergence can happen for a class of conic optimization problems, including semidefinite programs. Our proof techniques leverage deep connections with inexact ALM and primal-dual principles with PBM.

A Bundle-based Augmented Lagrangian Framework: Algorithm, Convergence, and Primal-dual Principles

TL;DR

This work introduces BALA, a single-loop bundle-based augmented Lagrangian method for constrained convex optimization, leveraging a bundle of past iterates to form a tractable inner approximation Ω_k ⊆ Ω and enabling exact subproblem solves. The authors establish sublinear convergence for primal feasibility, primal objective, and dual iterates, and prove linear convergence under quadratic growth and quadratic closeness of the dual approximation, with applicability to conic programs including semidefinite programs. The analysis reveals deep connections between BALA and the proximal bundle method (PBM) as well as inexact ALM, providing a unified primal–dual perspective. Numerical experiments on SDPs demonstrate that BALA outperforms conditional-gradient ALM variants, achieving high accuracy and exhibiting linear convergence in practice, highlighting its practical impact for large-scale conic optimization.

Abstract

We propose a new bundle-based augmented Lagrangian framework for solving constrained convex problems. Unlike the classical (inexact) augmented Lagrangian method (ALM) that has a nested double-loop structure, our framework features a process. Motivated by the proximal bundle method (PBM), we use a of past iterates to approximate the subproblem in ALM to get a computationally efficient update at each iteration. We establish sub-linear convergences for primal feasibility, primal cost values, and dual iterates under mild assumptions. With further regularity conditions, such as quadratic growth, our algorithm enjoys convergences. Importantly, this linear convergence can happen for a class of conic optimization problems, including semidefinite programs. Our proof techniques leverage deep connections with inexact ALM and primal-dual principles with PBM.

Paper Structure

This paper contains 41 sections, 19 theorems, 105 equations, 4 figures, 3 algorithms.

Table of Contents

  1. Introduction
  2. Preliminaries
  3. Augmented Lagrangian and its inexact version
  4. Proximal point method and its bundle version
  5. Our motivation of BALA
  6. A Bundle-based Augmented Lagrangian Framework and BALA
  7. A conceptual idea of inner approximations
  8. A novel bundled-based inner approximation
  9. Our algorithm: BALA
  10. Convergence Guarantees of BALA
  11. Progress of descent steps
  12. Sublinear convergences for convex problems
  13. Linear convergences under regularity conditions
  14. Numerical experiments
  15. Conclusion
  16. ...and 26 more sections

Key Result

Theorem 1

Suppose the Slater's condition holds for eq:primal. Let $\{x_k,y_k\}$ be a sequence from alg:alm with $\sum_{k=1}^\infty \epsilon_k < \infty$. Then, the sequence $\{y_k\}$ converges asymptotically to an optimal dual solution. Moreover, for all $k \geq 1$, we have which confirms the convergence of the primal affine feasibility and the cost value gap, i.e., we have $\lim_{k\to \infty} \|\mathcal{A}

Figures (4)

  • Figure 1: Comparison between our proposed BALA (\ref{['alg:bundle-dual']}) and CGALyurtsever2019conditionalyurtsever2021scalable.
  • Figure 2: Numerical experiment for a simple linear program
  • Figure 3: Linear convergence for SDPs.
  • Figure 4: Additional numerical experiments. The blue curves are from BALA, and the red curves are from CGALyurtsever2021scalable.

Theorems & Definitions (29)

  • Theorem 1: rockafellar1976augmented
  • Theorem 2: diaz2023optimal
  • Lemma 1: Implication of \ref{['eq:test']}
  • Lemma 2: Step length and primal residual
  • Lemma 3: Primal cost value
  • Lemma 4
  • Theorem 3: Asymptotic convergence
  • Theorem 4: Sublinear convergences
  • Theorem 5: Average iterates
  • Remark 4.1: Comparison with existing works
  • ...and 19 more