Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Subdifferentially polynomially bounded functions and Gaussian smoothing-based zeroth-order optimization

Ming Lei, Ting Kei Pong, Shuqin Sun, Man-Chung Yue

TL;DR

This work extends Gaussian smoothing based zeroth-order optimization to the broad class of subdifferentially polynomially bounded (SPB) functions, which includes Lipschitz, gradient-/Hessian-Lipschitz, and several nonsmooth models. It proves that GS is well-defined on SPB functions and that the GS gradient satisfies a descent-type relation with a polynomial-in-$\|x\|$ Lipschitz modulus, enabling adaptive-stepsize zeroth-order methods. The authors establish a precise link between the GS gradient and Goldstein $\delta$-subdifferentials, enabling approximate Goldstein stationarity to be characterized via GS gradients. They develop GS-based algorithms for convex and unconstrained SPB minimization, derive a descent-lemma-based complexity framework, and provide explicit iteration complexities for achieving $(\delta,\epsilon)$-approximate stationarity, offering practical guarantees for derivative-free optimization in non-Lipschitz settings.

Abstract

We study the class of subdifferentially polynomially bounded (SPB) functions, which is a rich class of locally Lipschitz functions that encompasses all Lipschitz functions, all gradient- or Hessian-Lipschitz functions, and even some non-smooth locally Lipschitz functions. We show that SPB functions are compatible with Gaussian smoothing (GS), in the sense that the GS of any SPB function is well-defined and satisfies a descent lemma akin to gradient-Lipschitz functions, with the Lipschitz constant replaced by a polynomial function. Leveraging this descent lemma, we propose GS-based zeroth-order optimization algorithms with an adaptive stepsize strategy for minimizing SPB functions, and analyze their convergence rates with respect to both relative and absolute stationarity measures. Finally, we also establish the iteration complexity for achieving a $(δ, ε)$-approximate stationary point, based on a novel quantification of Goldstein stationarity via the GS gradient that could be of independent interest.

Subdifferentially polynomially bounded functions and Gaussian smoothing-based zeroth-order optimization

TL;DR

This work extends Gaussian smoothing based zeroth-order optimization to the broad class of subdifferentially polynomially bounded (SPB) functions, which includes Lipschitz, gradient-/Hessian-Lipschitz, and several nonsmooth models. It proves that GS is well-defined on SPB functions and that the GS gradient satisfies a descent-type relation with a polynomial-in- Lipschitz modulus, enabling adaptive-stepsize zeroth-order methods. The authors establish a precise link between the GS gradient and Goldstein -subdifferentials, enabling approximate Goldstein stationarity to be characterized via GS gradients. They develop GS-based algorithms for convex and unconstrained SPB minimization, derive a descent-lemma-based complexity framework, and provide explicit iteration complexities for achieving -approximate stationarity, offering practical guarantees for derivative-free optimization in non-Lipschitz settings.

Abstract

We study the class of subdifferentially polynomially bounded (SPB) functions, which is a rich class of locally Lipschitz functions that encompasses all Lipschitz functions, all gradient- or Hessian-Lipschitz functions, and even some non-smooth locally Lipschitz functions. We show that SPB functions are compatible with Gaussian smoothing (GS), in the sense that the GS of any SPB function is well-defined and satisfies a descent lemma akin to gradient-Lipschitz functions, with the Lipschitz constant replaced by a polynomial function. Leveraging this descent lemma, we propose GS-based zeroth-order optimization algorithms with an adaptive stepsize strategy for minimizing SPB functions, and analyze their convergence rates with respect to both relative and absolute stationarity measures. Finally, we also establish the iteration complexity for achieving a -approximate stationary point, based on a novel quantification of Goldstein stationarity via the GS gradient that could be of independent interest.
Paper Structure (10 sections, 16 theorems, 132 equations, 2 algorithms)

This paper contains 10 sections, 16 theorems, 132 equations, 2 algorithms.

Table of Contents

  1. Introduction
  2. Notation and preliminaries
  3. Subdifferentially polynomially bounded functions
  4. Gaussian smoothing of SPB functions
  5. Approximate Goldstein stationarity
  6. GS-based algorithms for minimizing SPB functions
  7. Auxiliary lemmas for complexity analysis
  8. Convex SPB minimization
  9. Unconstrained SPB minimization
  10. Explicit complexity and ($\delta$, $\epsilon$)-stationarity

Key Result

Lemma 3.2

\newlabelprop:useful0 Let $f\in {{\@fontswitch{}{\mathcal{}} SPB}({\mathbb{R}}^d)}$ with parameters ${\rm R}_1$, ${\rm R}_2$ and $m$ as in definitioneq01. Then

Theorems & Definitions (39)

  • Definition 1.1: NesterovGS
  • Definition 3.1: Subdifferentially polynomially bounded functions
  • Example 3.1
  • Lemma 3.2
  • Proof 1
  • Theorem 3.3: Well-definedness of GS and its gradient
  • Proof 2
  • Theorem 3.4
  • Proof 3
  • Lemma 3.5
  • ...and 29 more