Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Stochastic Optimization with Random Search

El Mahdi Chayti, Taha El Bakkali El Kadi, Omar Saadi, Martin Jaggi

TL;DR

This work delivers a unified analysis of stochastic zeroth-order optimization via the Stochastic Three Points (STP) framework, showing that random search can achieve rates comparable to gradient-based methods under both standard and weaker smoothness assumptions. It extends STP to stochastic, finite-sum, and helper-driven feedback settings, introduces variance-reduction variants that improve complexity in the large-n regime, and leverages a translation invariance property to tightly control stochastic errors. The results reveal that weak average smoothness yields rates similar to prior stochastic STP, while standard per-component smoothness attains rates matching first-order methods, all without discretization bias. Empirically, the methods demonstrate competitive performance and illustrate that naive momentum adaptations do not improve Random Search, highlighting the need for novel variance-reduction or memory-efficient strategies in this paradigm.

Abstract

We revisit random search for stochastic optimization, where only noisy function evaluations are available. We show that the method works under weaker smoothness assumptions than previously considered, and that stronger assumptions enable improved guarantees. In the finite-sum setting, we design a variance-reduced variant that leverages multiple samples to accelerate convergence. Our analysis relies on a simple translation invariance property, which provides a principled way to balance noise and reduce variance.

Stochastic Optimization with Random Search

TL;DR

This work delivers a unified analysis of stochastic zeroth-order optimization via the Stochastic Three Points (STP) framework, showing that random search can achieve rates comparable to gradient-based methods under both standard and weaker smoothness assumptions. It extends STP to stochastic, finite-sum, and helper-driven feedback settings, introduces variance-reduction variants that improve complexity in the large-n regime, and leverages a translation invariance property to tightly control stochastic errors. The results reveal that weak average smoothness yields rates similar to prior stochastic STP, while standard per-component smoothness attains rates matching first-order methods, all without discretization bias. Empirically, the methods demonstrate competitive performance and illustrate that naive momentum adaptations do not improve Random Search, highlighting the need for novel variance-reduction or memory-efficient strategies in this paradigm.

Abstract

We revisit random search for stochastic optimization, where only noisy function evaluations are available. We show that the method works under weaker smoothness assumptions than previously considered, and that stronger assumptions enable improved guarantees. In the finite-sum setting, we design a variance-reduced variant that leverages multiple samples to accelerate convergence. Our analysis relies on a simple translation invariance property, which provides a principled way to balance noise and reduce variance.
Paper Structure (58 sections, 16 theorems, 130 equations, 1 figure, 1 algorithm)

This paper contains 58 sections, 16 theorems, 130 equations, 1 figure, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. General Algorithmic Framework
  4. Smoothness assumption
  5. Random directions
  6. Algorithm
  7. General descent bound
  8. Translation invariance
  9. General Stochastic Functions with Subsampling
  10. Case I: Weak average smoothness
  11. Case II: Standard sample smoothness
  12. Significance of the results
  13. Weak average smoothness.
  14. Standard sample smoothness.
  15. Finite-Sum Case and Variance Reduction
  16. ...and 43 more sections

Key Result

Lemma 3.1

Suppose $f$ is $(L_0,L_1)$-smooth and Assumption ass:distribution holds. If $\eta_t \leq \mu_{\mathcal{D}}/L_1$, then

Figures (1)

  • Figure 1: Breast Cancer (logistic):$f({\bm{x}})$ vs. queries for different batch sizes. Mean $\pm$ one s.d. across 20 runs.

Theorems & Definitions (27)

  • Lemma 3.1: Descent bound
  • Theorem 4.1: Average smoothness
  • Lemma 4.2: Difference approximation error
  • Theorem 4.3: Sample smoothness
  • Theorem 6.1: Convergence with $\delta$-inexact feedback
  • Lemma A.1: Variance of a minibatch average under bounded variance
  • proof
  • Lemma A.2: Jensen's inequality
  • proof
  • Lemma A.3: Adaptive Smoothness Inequality
  • ...and 17 more