Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Why Does Adaptive Zeroth-Order Optimization Work?

Haishan Ye, Luo Luo

TL;DR

This paper explains why adaptive zeroth-order optimization, which normalizes gradient estimates by the empirical standard deviation, is theoretically effective: with high probability, the standard deviation tracks the gradient norm, justifying adaptive step-size control. It introduces a generalized $(L_0,L_1)$-smoothness framework in the $\bm{H}$-norm, derives explicit convergence rates and query complexities for both deterministic and stochastic settings, and demonstrates that adaptive ZO achieves faster convergence and lower query costs than fixed-step vanilla ZO, including weak dimension dependence via sketching matrices. The proposed analysis, built on oblivious random sketching and rigorous event-based concentration, connects empirical practice with theory and extends ZO optimization to anisotropic, large-scale problems such as LLM fine-tuning. Practically, this work supports the use of standard deviation normalization in adaptive ZO as a principled, geometry-aware mechanism with provable efficiency gains. Overall, it lays a foundation for further exploration of adaptive, geometry-aware zeroth-order methods in nonconvex, high-dimensional settings.

Abstract

Zeroth-order (ZO) optimization is popular in real-world applications that accessing the gradient information is expensive or unavailable. Recently, adaptive ZO methods that normalize gradient estimators by the empirical standard deviation of function values have achieved strong practical performance, particularly in fine-tuning the large language model. However, the theoretical understanding of such strategy remains limited. In this work, we show that the empirical standard deviation is, with high probability, closely proportional to the norm of the (stochastic) gradient. Based on this insight, we analyze adaptive ZO methods under the generalized $(L_0,L_1)$-smoothness condition with respect to the matrix norm. We establish explicit convergence rates and query complexity bounds for both deterministic and stochastic settings, demonstrating that adaptive ZO methods achieve the faster convergence and the improved query efficiency compared to the vanilla ZO methods with fixed-step.

Why Does Adaptive Zeroth-Order Optimization Work?

TL;DR

This paper explains why adaptive zeroth-order optimization, which normalizes gradient estimates by the empirical standard deviation, is theoretically effective: with high probability, the standard deviation tracks the gradient norm, justifying adaptive step-size control. It introduces a generalized -smoothness framework in the -norm, derives explicit convergence rates and query complexities for both deterministic and stochastic settings, and demonstrates that adaptive ZO achieves faster convergence and lower query costs than fixed-step vanilla ZO, including weak dimension dependence via sketching matrices. The proposed analysis, built on oblivious random sketching and rigorous event-based concentration, connects empirical practice with theory and extends ZO optimization to anisotropic, large-scale problems such as LLM fine-tuning. Practically, this work supports the use of standard deviation normalization in adaptive ZO as a principled, geometry-aware mechanism with provable efficiency gains. Overall, it lays a foundation for further exploration of adaptive, geometry-aware zeroth-order methods in nonconvex, high-dimensional settings.

Abstract

Zeroth-order (ZO) optimization is popular in real-world applications that accessing the gradient information is expensive or unavailable. Recently, adaptive ZO methods that normalize gradient estimators by the empirical standard deviation of function values have achieved strong practical performance, particularly in fine-tuning the large language model. However, the theoretical understanding of such strategy remains limited. In this work, we show that the empirical standard deviation is, with high probability, closely proportional to the norm of the (stochastic) gradient. Based on this insight, we analyze adaptive ZO methods under the generalized -smoothness condition with respect to the matrix norm. We establish explicit convergence rates and query complexity bounds for both deterministic and stochastic settings, demonstrating that adaptive ZO methods achieve the faster convergence and the improved query efficiency compared to the vanilla ZO methods with fixed-step.
Paper Structure (20 sections, 36 theorems, 168 equations, 2 algorithms)

This paper contains 20 sections, 36 theorems, 168 equations, 2 algorithms.

Table of Contents

  1. Introduction
  2. Related Works
  3. Adaptive Zeroth-Order Method.
  4. Relaxed Smoothness.
  5. Preliminaries
  6. Notations
  7. Problem Settings
  8. Oblivious Random Sketching Matrix
  9. Algorithm Description of Adaptive Zeroth-Order Method
  10. Convergence Analysis for the Deterministic Problem
  11. Events
  12. Properties of Estimated Gradient and Standard Deviation
  13. Descent Analysis and Main Results
  14. Convergence Analysis for the Stochastic Problem
  15. Events
  16. ...and 5 more sections

Key Result

Lemma 1

Letting $\bm{S}_t$ be an oblivious $\left({1}/{4}, k, \delta\right)$-random sketching matrix follows Definition def:ske with $k\geq 4$ and $\delta\in(0,1)$, then event $\mathcal{E}_{t,1}$ holds with a probability at least $1-\delta$ with $\zeta_1$ defined as

Theorems & Definitions (74)

  • Definition 1: Sketching in Matrix Product
  • Lemma 1
  • proof
  • Lemma 2
  • proof
  • Lemma 3
  • proof
  • Lemma 4
  • proof
  • Lemma 5
  • ...and 64 more