Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

A Study of Condition Numbers for First-Order Optimization

Charles Guille-Escuret, Baptiste Goujaud, Manuela Girotti, Ioannis Mitliagkas

TL;DR

The paper argues that classical tuning of first-order optimization algorithms based on $L$-smoothness and $\mu$-strong convexity is brittle under small perturbations of the objective. It introduces a star-norm $\|\cdot\|_*$ to measure perturbation impact and establishes a continuity framework for FOA behavior under these perturbations, showing that FOA trajectories are robust for small star-norm changes while traditional metrics are not. To build robust tuning, it defines continuous upper and lower condition families and demonstrates their relationships, including convergence guarantees for Gradient Descent across various condition-pair combinations. It then advocates for using these continuous conditions (and their implications for FOA tuning and convergence) in practice, illustrated by potential improvements in HB behavior on challenging functions and connections to logistic regression settings.

Abstract

The study of first-order optimization algorithms (FOA) typically starts with assumptions on the objective functions, most commonly smoothness and strong convexity. These metrics are used to tune the hyperparameters of FOA. We introduce a class of perturbations quantified via a new norm, called *-norm. We show that adding a small perturbation to the objective function has an equivalently small impact on the behavior of any FOA, which suggests that it should have a minor impact on the tuning of the algorithm. However, we show that smoothness and strong convexity can be heavily impacted by arbitrarily small perturbations, leading to excessively conservative tunings and convergence issues. In view of these observations, we propose a notion of continuity of the metrics, which is essential for a robust tuning strategy. Since smoothness and strong convexity are not continuous, we propose a comprehensive study of existing alternative metrics which we prove to be continuous. We describe their mutual relations and provide their guaranteed convergence rates for the Gradient Descent algorithm accordingly tuned. Finally we discuss how our work impacts the theoretical understanding of FOA and their performances.

A Study of Condition Numbers for First-Order Optimization

TL;DR

The paper argues that classical tuning of first-order optimization algorithms based on -smoothness and -strong convexity is brittle under small perturbations of the objective. It introduces a star-norm to measure perturbation impact and establishes a continuity framework for FOA behavior under these perturbations, showing that FOA trajectories are robust for small star-norm changes while traditional metrics are not. To build robust tuning, it defines continuous upper and lower condition families and demonstrates their relationships, including convergence guarantees for Gradient Descent across various condition-pair combinations. It then advocates for using these continuous conditions (and their implications for FOA tuning and convergence) in practice, illustrated by potential improvements in HB behavior on challenging functions and connections to logistic regression settings.

Abstract

The study of first-order optimization algorithms (FOA) typically starts with assumptions on the objective functions, most commonly smoothness and strong convexity. These metrics are used to tune the hyperparameters of FOA. We introduce a class of perturbations quantified via a new norm, called *-norm. We show that adding a small perturbation to the objective function has an equivalently small impact on the behavior of any FOA, which suggests that it should have a minor impact on the tuning of the algorithm. However, we show that smoothness and strong convexity can be heavily impacted by arbitrarily small perturbations, leading to excessively conservative tunings and convergence issues. In view of these observations, we propose a notion of continuity of the metrics, which is essential for a robust tuning strategy. Since smoothness and strong convexity are not continuous, we propose a comprehensive study of existing alternative metrics which we prove to be continuous. We describe their mutual relations and provide their guaranteed convergence rates for the Gradient Descent algorithm accordingly tuned. Finally we discuss how our work impacts the theoretical understanding of FOA and their performances.

Paper Structure

This paper contains 53 sections, 7 theorems, 147 equations, 2 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Setup and notation
  4. Continuity of first-order algorithms and condition numbers
  5. Star norm and stability of FOA behaviors
  6. Standard tuning fails continuity test
  7. Continuity of condition numbers
  8. Alternative conditioning
  9. Gradient descent convergence
  10. Discussion
  11. Conclusion
  12. Acknowledgements.
  13. Continuity
  14. Proof of Theorem \ref{['continuity1']}
  15. Proof of Corollary \ref{['continuity3']}
  16. ...and 38 more sections

Key Result

Theorem 4.5

Let $f\in C^1(\mathbb{R}^d)$ with a set of global minimizers $X^*$ and $\left\|\cdot\right\|_*$ the corresponding star norm. Let $\mathcal{A}_\theta$ be a continuous first-order algorithm and $\mathcal{K}\subset \mathbb{R}^d$ a compact set. Then, the following result holds:

Figures (2)

  • Figure 1: Graph of implications between upper and lower conditions. Red arrows only hold under $^*\!\operatorname{SC}^-(\mu)$ or $\operatorname{SC}^-(\mu)$ where $\mu$ can be negative. Green arrows only hold under $\operatorname{QG}^+$ or $\operatorname{QG}^-$.
  • Figure 2: Convergence rate of HB on $f_{\rm LRP}$ with starting point $x_0 = 3.3$ for different tunings of $\alpha$, $\beta$. Black areas mean no linear convergence

Theorems & Definitions (23)

  • Definition 3.1: Continuous FOA
  • Definition 4.1: Star norm
  • Example 4.2
  • Example 4.3
  • Remark 4.4
  • Theorem 4.5
  • Corollary 4.6
  • Proposition 4.7
  • Definition 4.8: Upper conditions
  • Definition 4.9: Lower conditions
  • ...and 13 more