Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Improved Analysis of Restarted Accelerated Gradient and Augmented Lagrangian Methods via Inexact Proximal Point Frameworks

Matthew X. Burns, Jiaming Liang

TL;DR

A restarted accelerated composite gradient method is developed that attains the optimal first-order complexity in both the convex and strongly convex settings and establishes near-optimal first-order complexity for both methods.

Abstract

This paper studies a class of double-loop (inner-outer) algorithms for convex composite optimization. For unconstrained problems, we develop a restarted accelerated composite gradient method that attains the optimal first-order complexity in both the convex and strongly convex settings. For linearly constrained problems, we introduce inexact augmented Lagrangian methods, including a basic method and an outer-accelerated variant, and establish near-optimal first-order complexity for both methods. The established complexity bounds follow from a unified analysis based on new inexact proximal point frameworks that accommodate relative and absolute inexactness, acceleration, and strongly convex objectives. Numerical experiments on LASSO and linearly constrained quadratic programs demonstrate the practical efficiency of the proposed methods.

Improved Analysis of Restarted Accelerated Gradient and Augmented Lagrangian Methods via Inexact Proximal Point Frameworks

TL;DR

A restarted accelerated composite gradient method is developed that attains the optimal first-order complexity in both the convex and strongly convex settings and establishes near-optimal first-order complexity for both methods.

Abstract

This paper studies a class of double-loop (inner-outer) algorithms for convex composite optimization. For unconstrained problems, we develop a restarted accelerated composite gradient method that attains the optimal first-order complexity in both the convex and strongly convex settings. For linearly constrained problems, we introduce inexact augmented Lagrangian methods, including a basic method and an outer-accelerated variant, and establish near-optimal first-order complexity for both methods. The established complexity bounds follow from a unified analysis based on new inexact proximal point frameworks that accommodate relative and absolute inexactness, acceleration, and strongly convex objectives. Numerical experiments on LASSO and linearly constrained quadratic programs demonstrate the practical efficiency of the proposed methods.
Paper Structure (37 sections, 35 theorems, 218 equations, 2 figures, 1 table, 6 algorithms)

This paper contains 37 sections, 35 theorems, 218 equations, 2 figures, 1 table, 6 algorithms.

Table of Contents

  1. Introduction
  2. Basic Definitions and Notation
  3. Primal Algorithm: Restarted ACG
  4. Overview of an ACG variant
  5. The Restarted ACG Method
  6. Dual Algorithm: Augmented Lagrangian
  7. Inexact Augmented Lagrangian Method
  8. Inexact Fast Augmented Lagrangian Method
  9. Frameworks for Generic Convex Optimization
  10. Lower Oracle Approximation Framework
  11. Fast Lower Oracle Approximation Framework
  12. Proofs of Main Complexity Results
  13. Proof of Theorem \ref{['thm:main']}
  14. Proof of Theorem \ref{['thm:al_baseline_complexity']}
  15. Proof of Theorem \ref{['thm:acc_alm_complexity_1']}
  16. ...and 22 more sections

Key Result

Lemma 2.1

Define $R_0=\min\{\|x-x_0\|:x\in X_*\}$, where $X_*$ is the set of optimal solutions to eq:prob. Then, for all $j\geq 1$,

Figures (2)

  • Figure 1: Numerical results for Restarted ACG algorithms.
  • Figure 2: Numerical experiments for ALM variants tested.

Theorems & Definitions (51)

  • Lemma 2.1
  • Lemma 2.2
  • Theorem 2.3
  • Theorem 3.1
  • Corollary 3.2
  • Lemma 3.3: Perturbation Solution
  • Theorem 3.4
  • Corollary 3.5
  • Proposition 4.1
  • Example 4.2: Proximal Gradient Method
  • ...and 41 more