Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Computer-Assisted Design of Accelerated Composite Optimization Methods: OptISTA

Uijeong Jang, Shuvomoy Das Gupta, Ernest K. Ryu

TL;DR

This work presents a double-function stepsize-optimization PEP methodology that poses the optimization over fixed-step first-order methods for composite optimization as a finite-dimensional nonconvex QCQP, which can be practically solved through spatial branch-and-bound algorithms, and designs the exact optimal method OptISTA for the composite optimization setup.

Abstract

The accelerated composite optimization method FISTA (Beck, Teboulle 2009) is suboptimal by a constant factor, and we present a new method OptISTA that improves FISTA by a constant factor of 2. The performance estimation problem (PEP) has recently been introduced as a new computer-assisted paradigm for designing optimal first-order methods. In this work, we present a double-function stepsize-optimization PEP methodology that poses the optimization over fixed-step first-order methods for composite optimization as a finite-dimensional nonconvex QCQP, which can be practically solved through spatial branch-and-bound algorithms, and use it to design the exact optimal method OptISTA for the composite optimization setup. We then establish the exact optimality of OptISTA under the large-scale assumption with a lower-bound construction that extends the semi-interpolated zero-chain construction (Drori, Taylor 2022) to the double-function setup of composite optimization. By establishing exact optimality, our work concludes the search for the fastest first-order methods, with respect to the performance measure of worst-case function value suboptimality, for the proximal, projected-gradient, and proximal-gradient setups involving a smooth convex function and a closed proper convex function.

Computer-Assisted Design of Accelerated Composite Optimization Methods: OptISTA

TL;DR

This work presents a double-function stepsize-optimization PEP methodology that poses the optimization over fixed-step first-order methods for composite optimization as a finite-dimensional nonconvex QCQP, which can be practically solved through spatial branch-and-bound algorithms, and designs the exact optimal method OptISTA for the composite optimization setup.

Abstract

The accelerated composite optimization method FISTA (Beck, Teboulle 2009) is suboptimal by a constant factor, and we present a new method OptISTA that improves FISTA by a constant factor of 2. The performance estimation problem (PEP) has recently been introduced as a new computer-assisted paradigm for designing optimal first-order methods. In this work, we present a double-function stepsize-optimization PEP methodology that poses the optimization over fixed-step first-order methods for composite optimization as a finite-dimensional nonconvex QCQP, which can be practically solved through spatial branch-and-bound algorithms, and use it to design the exact optimal method OptISTA for the composite optimization setup. We then establish the exact optimality of OptISTA under the large-scale assumption with a lower-bound construction that extends the semi-interpolated zero-chain construction (Drori, Taylor 2022) to the double-function setup of composite optimization. By establishing exact optimality, our work concludes the search for the fastest first-order methods, with respect to the performance measure of worst-case function value suboptimality, for the proximal, projected-gradient, and proximal-gradient setups involving a smooth convex function and a closed proper convex function.
Paper Structure (60 sections, 56 theorems, 420 equations)

This paper contains 60 sections, 56 theorems, 420 equations.

Table of Contents

  1. Introduction
  2. Preliminaries and notation.
  3. Prior work
  4. FISTA and its variants.
  5. Proximal point method and its variants.
  6. Lower bounds.
  7. Performance estimation problem: A foundation for optimal methods.
  8. Contributions
  9. New exact optimal method
  10. OptISTA: Exact optimal composite optimization method
  11. OptISTA proof outline
  12. Computer-assisted algorithmic design via PEP
  13. Double-function fixed-step first-order methods
  14. Discussion on stepsize dependence.
  15. Double-function stepsize-optimization PEP (DF-SO-PEP)
  16. ...and 45 more sections

Key Result

theorem 1

Let $f\colon {\mathbb{R}}^{d}\to{\mathbb{R}}$ be $L$-smooth convex and $h\colon{\mathbb{R}}^{d}\to{\mathbb{R}}\cup\{\infty\}$ be closed, convex, and proper. Assume $x_\star\in \mathop{{\rm argmin}}(f+h)$ exists. Let $N>0$. Then, eq:OptISTA_Alg exhibits the rate where $\theta_{N}$ is as defined for eq:OptISTA_Alg.

Theorems & Definitions (111)

  • theorem 1
  • Lemma 1
  • proof
  • Lemma 2
  • proof
  • Lemma 3: $\mathcal{F}_{0,L}$- and $\mathcal{F}_{0,\infty}$-interpolation taylor2017InterpolationFMuLPEP
  • Lemma 4: horn2012matrix
  • theorem 2
  • Lemma 5
  • Lemma 6
  • ...and 101 more