Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Lipschitz minimization and the Goldstein modulus

Siyu Kong, Adrian S. Lewis

TL;DR

The paper introduces a robust slope measure for Lipschitz objectives, the Goldstein modulus $\Gamma f$, and shows that linear growth of this modulus near a minimizer can explain near\-linear convergence of Goldstein style subgradient methods. It develops a cohesive theory linking growth conditions, manifold structure, and max function geometry to ensure linear growth of $\Gamma f$ and fast convergence, including results for robust growth relative to a manifold and for strong smooth max representations. A simple idealized descent is analyzed to yield a sublinear $O(\log s / s)$ rate under linear modulus growth, while stronger structural assumptions yield nearly linear convergence with $|x_r-\bar{x}|$ decaying as $e^{-\beta\sqrt{r}}$. The work further provides tempered growth results and a practical heuristic that approximates Goldstein subgradients via randomized refinements, offering actionable guidance for implementing Lipschitz minimization in nonsmooth settings and illuminating the role of objective geometry in convergence.

Abstract

Goldstein's 1977 idealized iteration for minimizing a Lipschitz objective fixes a distance - the step size - and relies on a certain approximate subgradient. That "Goldstein subgradient" is the shortest convex combination of objective gradients at points within that distance of the current iterate. A recent implementable Goldstein-style algorithm allows a remarkable complexity analysis (Zhang et al. 2020), and a more sophisticated variant (Davis and Jiang, 2022) leverages typical objective geometry to force near-linear convergence. To explore such methods, we introduce a new modulus, based on Goldstein subgradients, that robustly measures the slope of a Lipschitz function. We relate near-linear convergence of Goldstein-style methods to linear growth of this modulus at minimizers. We illustrate the idea computationally with a simple heuristic for Lipschitz minimization.

Lipschitz minimization and the Goldstein modulus

TL;DR

The paper introduces a robust slope measure for Lipschitz objectives, the Goldstein modulus , and shows that linear growth of this modulus near a minimizer can explain near\-linear convergence of Goldstein style subgradient methods. It develops a cohesive theory linking growth conditions, manifold structure, and max function geometry to ensure linear growth of and fast convergence, including results for robust growth relative to a manifold and for strong smooth max representations. A simple idealized descent is analyzed to yield a sublinear rate under linear modulus growth, while stronger structural assumptions yield nearly linear convergence with decaying as . The work further provides tempered growth results and a practical heuristic that approximates Goldstein subgradients via randomized refinements, offering actionable guidance for implementing Lipschitz minimization in nonsmooth settings and illuminating the role of objective geometry in convergence.

Abstract

Goldstein's 1977 idealized iteration for minimizing a Lipschitz objective fixes a distance - the step size - and relies on a certain approximate subgradient. That "Goldstein subgradient" is the shortest convex combination of objective gradients at points within that distance of the current iterate. A recent implementable Goldstein-style algorithm allows a remarkable complexity analysis (Zhang et al. 2020), and a more sophisticated variant (Davis and Jiang, 2022) leverages typical objective geometry to force near-linear convergence. To explore such methods, we introduce a new modulus, based on Goldstein subgradients, that robustly measures the slope of a Lipschitz function. We relate near-linear convergence of Goldstein-style methods to linear growth of this modulus at minimizers. We illustrate the idea computationally with a simple heuristic for Lipschitz minimization.
Paper Structure (8 sections, 17 theorems, 103 equations, 3 figures)

This paper contains 8 sections, 17 theorems, 103 equations, 3 figures.

Table of Contents

  1. Introduction: the Goldstein subgradient
  2. The Goldstein modulus
  3. A simple Goldstein descent algorithm
  4. Nearly linear convergence
  5. Robust growth relative to a manifold
  6. Robust growth for max functions
  7. Tempered growth relative to a manifold
  8. A Goldstein-style heuristic

Key Result

Proposition 2.1

For any $L$-Lipschitz objective function $f \colon {\mathbf X} \to {\mathbf R}$, any point $x \in {\mathbf X}$, and any radius $\epsilon \ge 0$, the Goldstein subgradient satisfies $|g_\epsilon(x)| \le L$.

Figures (3)

  • Figure 1: Algorithm \ref{['min-for-lip']} minimizing a maximum of five nonconvex quadratics on ${\mathbf R}^{10}$, using an approximate Goldstein subgradient oracle, illustrating near-linear convergence of the iterates to the minimizer.
  • Figure 2: The example of Figure \ref{['fig1']}, illustrating convergence of the objective value.
  • Figure 3: The example of Figure \ref{['fig1']}, illustrating convergence with respect to subgradient evaluations.

Theorems & Definitions (29)

  • Proposition 2.1
  • Definition 2.2
  • Proposition 2.4
  • Proposition 2.6
  • Theorem 3.2: Convergence rate
  • Example 3.3
  • Definition 4.1
  • Theorem 4.4: Near-linear convergence
  • Definition 4.5
  • Corollary 4.6
  • ...and 19 more