Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

ripALM: A Relative-Type Inexact Proximal Augmented Lagrangian Method for Linearly Constrained Convex Optimization

Jiayi Zhu, Ling Liang, Lei Yang, Kim-Chuan Toh

TL;DR

The paper tackles linearly constrained convex optimization by enhancing the proximal augmented Lagrangian framework with a relative-type inexactness criterion, yielding ripALM. It presents a dual-based formulation with a proximal term, proving convergence of primal and dual sequences and establishing potential for asymptotic (super)linear rates under a mild error-bound assumption. The authors demonstrate competitive performance on quadratically regularized OT and basis pursuit denoising, outperforming several absolute- and relative-type competitors and standard solvers in many settings. The work suggests broad applicability and highlights practical advantages in implementation and robustness for large-scale convex optimization tasks.

Abstract

Inexact proximal augmented Lagrangian methods (pALMs) are particularly appealing for tackling convex constrained optimization problems because of their elegant convergence properties and strong practical performance. To solve the associated pALM subproblems, efficient methods such as Newton-type methods are essential. Consequently, the effectiveness of the inexact pALM hinges on the error criteria used to control the inexactness when solving these subproblems. However, existing inexact pALMs either rely on \textit{absolute-type} error criteria (which may complicate implementation by necessitating the pre-specification of an infinite sequence of error tolerance parameters) or require an additional correction step to guarantee convergence when using \textit{relative-type} error criteria (which can potentially degrade the practical performance of pALM). To address these deficiencies, this paper proposes ripALM, a relative-type inexact pALM, for linearly constrained convex optimization. This method can simplify practical implementation while preserving the appealing convergence properties of the classical absolute-type inexact pALM. To the best of our knowledge, ripALM is the first relative-type inexact version of the vanilla pALM with provable convergence guarantees. Numerical experiments on quadratically regularized optimal transport problems and basis pursuit denoising problems demonstrate the competitive efficiency of the proposed method compared to existing state-of-the-art methods. As our model encompasses a wide range of application problems, the proposed ripALM offers broad applicability and has the potential to serve as a basic optimization toolbox.

ripALM: A Relative-Type Inexact Proximal Augmented Lagrangian Method for Linearly Constrained Convex Optimization

TL;DR

The paper tackles linearly constrained convex optimization by enhancing the proximal augmented Lagrangian framework with a relative-type inexactness criterion, yielding ripALM. It presents a dual-based formulation with a proximal term, proving convergence of primal and dual sequences and establishing potential for asymptotic (super)linear rates under a mild error-bound assumption. The authors demonstrate competitive performance on quadratically regularized OT and basis pursuit denoising, outperforming several absolute- and relative-type competitors and standard solvers in many settings. The work suggests broad applicability and highlights practical advantages in implementation and robustness for large-scale convex optimization tasks.

Abstract

Inexact proximal augmented Lagrangian methods (pALMs) are particularly appealing for tackling convex constrained optimization problems because of their elegant convergence properties and strong practical performance. To solve the associated pALM subproblems, efficient methods such as Newton-type methods are essential. Consequently, the effectiveness of the inexact pALM hinges on the error criteria used to control the inexactness when solving these subproblems. However, existing inexact pALMs either rely on \textit{absolute-type} error criteria (which may complicate implementation by necessitating the pre-specification of an infinite sequence of error tolerance parameters) or require an additional correction step to guarantee convergence when using \textit{relative-type} error criteria (which can potentially degrade the practical performance of pALM). To address these deficiencies, this paper proposes ripALM, a relative-type inexact pALM, for linearly constrained convex optimization. This method can simplify practical implementation while preserving the appealing convergence properties of the classical absolute-type inexact pALM. To the best of our knowledge, ripALM is the first relative-type inexact version of the vanilla pALM with provable convergence guarantees. Numerical experiments on quadratically regularized optimal transport problems and basis pursuit denoising problems demonstrate the competitive efficiency of the proposed method compared to existing state-of-the-art methods. As our model encompasses a wide range of application problems, the proposed ripALM offers broad applicability and has the potential to serve as a basic optimization toolbox.

Paper Structure

This paper contains 19 sections, 2 theorems, 129 equations, 1 figure, 6 tables, 4 algorithms.

Table of Contents

  1. Introduction
  2. A relative-type inexact pALM
  3. Convergence analysis
  4. Applications of ripALM
  5. Quadratically regularized optimal transport
  6. Basis pursuit denoising
  7. Numerical experiments
  8. Comparison with SNIPAL and ciPALM
  9. Comparison with Gurobi, dADMM blueand ES-iALM
  10. Experiments for BPDN
  11. Conclusions
  12. Missing Proofs in Section \ref{['sec:conver']}
  13. Proof of Theorem \ref{['thm:convergence']}
  14. Proof of Theorem \ref{['thm:Q-linear-rate']}
  15. A dual ADMM for QROT
  16. ...and 4 more sections

Key Result

Theorem 3.1

Let the functions $G$, $F$ and $\ell$ be defined as in eq:para-objpri, eq:para-objdual and eq:ripALM_lag, respectively. Let $\rho\in[0,1)$, $\{\sigma_{k}\}$ be a positive sequence satisfying that $\sigma_k\geq\sigma_{\min}>0$ for all $k\geq0$, and $\{\tau_{k}\}$ be a positive sequence satisfying tha Let $\{\bm{y}^{k}\}$, $\{\Delta^k\}$, $\{\bm{w}^k\}\subset\mathbb{R}^{M}$ and $\{\bm{x}^{k}\}\subse

Figures (1)

  • Figure 1: Comparisons among Gurobi, dADMM, ES-iALM, and ripALM for solving the QROT problem \ref{['QROTprob']} with $m = n\in\{1000,\,2000,\cdots,10000\}$. The shaded region indicates the maximum and minimum computational times taken to solve 10 instances for each problem dimension.

Theorems & Definitions (7)

  • Theorem 3.1
  • proof
  • Theorem 3.2
  • proof
  • Remark 3.1: Comments on blue$\gamma_k$ and $\mu_{k}$
  • proof
  • proof