Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

A Parameter-Free and Near-Optimal Zeroth-Order Algorithm for Stochastic Convex Optimization

Kunjie Ren, Luo Luo

TL;DR

This work tackles stochastic convex optimization under zeroth-order access by introducing POEM, a parameter-free method that jointly tunes the step size and the smoothing parameter using distance-based quantities. The approach relies on randomized smoothing to form a differentiable surrogate and a finite-difference gradient estimator, with step sizes scaled by the distance from the initialization and cumulative gradient magnitude. The authors prove near-optimal stochastic zeroth-order complexity, with an overall rate of ${\tilde{\mathcal O}}(d L^2 D_{\mathcal X}^2 / \epsilon^2)$, and they provide high-probability guarantees; they also analyze the unbounded-domain case, proving the impossibility of ideal parameter-free design there. Numerical experiments demonstrate that POEM outperforms standard zeroth-order methods and is robust to tuning, supporting its practical impact for black-box stochastic convex optimization.

Abstract

This paper considers zeroth-order optimization for stochastic convex minimization problem. We propose a parameter-free stochastic zeroth-order method (POEM) by introducing a step-size scheme based on the distance over finite difference and an adaptive smoothing parameter. We provide the theoretical analysis to show that POEM achieves the near-optimal stochastic zeroth-order oracle complexity. We further conduct the numerical experiments to demonstrate POEM outperforms existing zeroth-order methods in practice.

A Parameter-Free and Near-Optimal Zeroth-Order Algorithm for Stochastic Convex Optimization

TL;DR

This work tackles stochastic convex optimization under zeroth-order access by introducing POEM, a parameter-free method that jointly tunes the step size and the smoothing parameter using distance-based quantities. The approach relies on randomized smoothing to form a differentiable surrogate and a finite-difference gradient estimator, with step sizes scaled by the distance from the initialization and cumulative gradient magnitude. The authors prove near-optimal stochastic zeroth-order complexity, with an overall rate of , and they provide high-probability guarantees; they also analyze the unbounded-domain case, proving the impossibility of ideal parameter-free design there. Numerical experiments demonstrate that POEM outperforms standard zeroth-order methods and is robust to tuning, supporting its practical impact for black-box stochastic convex optimization.

Abstract

This paper considers zeroth-order optimization for stochastic convex minimization problem. We propose a parameter-free stochastic zeroth-order method (POEM) by introducing a step-size scheme based on the distance over finite difference and an adaptive smoothing parameter. We provide the theoretical analysis to show that POEM achieves the near-optimal stochastic zeroth-order oracle complexity. We further conduct the numerical experiments to demonstrate POEM outperforms existing zeroth-order methods in practice.

Paper Structure

This paper contains 21 sections, 25 theorems, 125 equations, 3 figures, 1 table, 1 algorithm.

Table of Contents

  1. Introduction
  2. Preliminaries
  3. Notation and Assumptions
  4. Randomized Smoothing
  5. Parameter-Free Stochastic Zeroth-Order Optimization
  6. The Complexity Analysis
  7. Results for Unbounded Domains
  8. Numerical Experiments
  9. Conclusion
  10. Some Basic Results
  11. The Proofs for Section \ref{['sec:complexity']}
  12. Proof of Lemma \ref{['lem:upper-bound-for-g']}
  13. Proof of Lemma \ref{['lem:noise-from-g']}
  14. Proof of Lemma \ref{['lem:noise-from-mu']}
  15. Proof of Proposition \ref{['prop']}
  16. ...and 6 more sections

Key Result

Lemma 1

Under Assumptions asm:convex and asm:lipschitz, the smooth surrogate $f_\mu({\bf{x}})$ is convex and satisfies $|f_\mu({\bf{x}})-f({\bf{x}})|\leq L\mu$ for all ${\bf{x}}\in{\mathbb{R}}^d$.

Figures (3)

  • Figure 1: The comparison on the SZO complexity against the function value during the iterations.
  • Figure 2: The comparison on parameter settings ($r_\epsilon$ for POEM and $1/L$ for other methods) against $f({\bf{x}}_T)$.
  • Figure 3: The change of the step size with difference $r_\epsilon$ for POEM.

Theorems & Definitions (46)

  • Definition 1
  • Definition 2
  • Definition 3
  • Lemma 1: shamir2017optimal
  • Lemma 2: flaxman2004online
  • Lemma 3: ivgi2023dog
  • Lemma 4: shamir2017optimal
  • Remark 1
  • Lemma 5
  • Lemma 6
  • ...and 36 more