Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Optimal First-Order Algorithms as a Function of Inequalities

Chanwoo Park, Ernest K. Ryu

TL;DR

This work presents a novel algorithm design methodology that restricts convergence analyses of algorithms to use a prespecified subset of inequalities, rather than utilizing all true inequalities, and finds the optimal algorithm subject to this restriction.

Abstract

In this work, we present a novel algorithm design methodology that finds the optimal algorithm as a function of inequalities. Specifically, we restrict convergence analyses of algorithms to use a prespecified subset of inequalities, rather than utilizing all true inequalities, and find the optimal algorithm subject to this restriction. This methodology allows us to design algorithms with certain desired characteristics. As concrete demonstrations of this methodology, we find new state-of-the-art accelerated first-order gradient methods using randomized coordinate updates and backtracking line searches.

Optimal First-Order Algorithms as a Function of Inequalities

TL;DR

This work presents a novel algorithm design methodology that restricts convergence analyses of algorithms to use a prespecified subset of inequalities, rather than utilizing all true inequalities, and finds the optimal algorithm subject to this restriction.

Abstract

In this work, we present a novel algorithm design methodology that finds the optimal algorithm as a function of inequalities. Specifically, we restrict convergence analyses of algorithms to use a prespecified subset of inequalities, rather than utilizing all true inequalities, and find the optimal algorithm subject to this restriction. This methodology allows us to design algorithms with certain desired characteristics. As concrete demonstrations of this methodology, we find new state-of-the-art accelerated first-order gradient methods using randomized coordinate updates and backtracking line searches.

Paper Structure

This paper contains 48 sections, 9 theorems, 274 equations.

Table of Contents

  1. Introduction
  2. Contributions.
  3. Preliminaries and notations
  4. Problem setting and $L$-smoothness.
  5. Fixed-step first-order algorithms (FSFO).
  6. Nesterov's fast gradient method (FGM).
  7. Optimized gradient method (OGM).
  8. Optimized gradient method - Gradient norm (OGM-G).
  9. Coordinate-wise smoothness.
  10. Fast gradient method - randomized coordinate updates (FGM-RC).
  11. Fast gradient method - backtracking linesearch (FGM-BL).
  12. Computer-assisted algorithm design.
  13. Organization
  14. Handy inequalities for deriving variants of FSOM
  15. Inequalities for smooth convex functions
  16. ...and 33 more sections

Key Result

Theorem 1

ORC-F$_\flat$ is $\mathcal{A}^\star$-optimal in the sense that and has the minimax optimal rate with respect to the inequalities

Theorems & Definitions (22)

  • Theorem 1: $\mathcal{A}^\star$-optimality of ORC-F$_\flat$
  • Corollary 2
  • Theorem 3
  • Claim 1
  • Claim 2
  • Claim 3
  • Claim 4
  • Theorem 4: $\mathcal{A}^\star$-optimality of OBL-F$_\flat$
  • Corollary 5
  • Theorem 6
  • ...and 12 more