Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

ZIVR: An Incremental Variance Reduction Technique For Zeroth-Order Composite Problems

Silan Zhang, Yujie Tang

TL;DR

This work tackles zeroth-order finite-sum composite optimization where gradient information is unavailable and standard first-order variance reduction cannot be directly applied. It introduces ZIVR, a general framework that maintains an incremental Jacobian estimator $J_k$ to construct a variance-reduced gradient $g_k$ by blending historical gradient information with current two-point zeroth-order estimates, enabling pure $2$-point updates and memory-efficient variants. The authors prove comprehensive convergence guarantees for strongly convex, convex, and non-convex regimes, achieving rates that match first-order methods up to a factor of $O(d)$ and providing oracle-complexity bounds that scale favorably with problem dimensions. They validate the approach through numerical experiments on logistic and Cox regression, showing that ZIVR outperforms competing zeroth-order methods while reducing batch-size requirements. Overall, ZIVR offers a scalable and principled way to perform constrained or composite zeroth-order optimization with provable efficiency guarantees.

Abstract

This paper investigates zeroth-order (ZO) finite-sum composite optimization. Recently, variance reduction techniques have been applied to ZO methods to mitigate the non-vanishing variance of 2-point estimators in constrained/composite optimization, yielding improved convergence rates. However, existing ZO variance reduction methods typically involve batch sampling of size at least $Θ(n)$ or $Θ(d)$, which can be computationally prohibitive for large-scale problems. In this work, we propose a general variance reduction framework, Zeroth-Order Incremental Variance Reduction (ZIVR), which supports flexible implementations$\unicode{x2014}$including a pure 2-point zeroth-order algorithm that eliminates the need for large batch sampling. Furthermore, we establish comprehensive convergence guarantees for ZIVR across strongly-convex, convex, and non-convex settings that match their first-order counterparts. Numerical experiments validate the effectiveness of our proposed algorithm.

ZIVR: An Incremental Variance Reduction Technique For Zeroth-Order Composite Problems

TL;DR

This work tackles zeroth-order finite-sum composite optimization where gradient information is unavailable and standard first-order variance reduction cannot be directly applied. It introduces ZIVR, a general framework that maintains an incremental Jacobian estimator to construct a variance-reduced gradient by blending historical gradient information with current two-point zeroth-order estimates, enabling pure -point updates and memory-efficient variants. The authors prove comprehensive convergence guarantees for strongly convex, convex, and non-convex regimes, achieving rates that match first-order methods up to a factor of and providing oracle-complexity bounds that scale favorably with problem dimensions. They validate the approach through numerical experiments on logistic and Cox regression, showing that ZIVR outperforms competing zeroth-order methods while reducing batch-size requirements. Overall, ZIVR offers a scalable and principled way to perform constrained or composite zeroth-order optimization with provable efficiency guarantees.

Abstract

This paper investigates zeroth-order (ZO) finite-sum composite optimization. Recently, variance reduction techniques have been applied to ZO methods to mitigate the non-vanishing variance of 2-point estimators in constrained/composite optimization, yielding improved convergence rates. However, existing ZO variance reduction methods typically involve batch sampling of size at least or , which can be computationally prohibitive for large-scale problems. In this work, we propose a general variance reduction framework, Zeroth-Order Incremental Variance Reduction (ZIVR), which supports flexible implementationsincluding a pure 2-point zeroth-order algorithm that eliminates the need for large batch sampling. Furthermore, we establish comprehensive convergence guarantees for ZIVR across strongly-convex, convex, and non-convex settings that match their first-order counterparts. Numerical experiments validate the effectiveness of our proposed algorithm.
Paper Structure (36 sections, 19 theorems, 129 equations, 2 figures, 2 tables, 2 algorithms)

This paper contains 36 sections, 19 theorems, 129 equations, 2 figures, 2 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Main Contributions
  3. Notations.
  4. Problem Formulation
  5. Algorithm
  6. Jacobian Estimator $J_k$
  7. Gradient Estimator $g_k$
  8. Examples of Implementations
  9. Implementation I
  10. Implementation II
  11. Implementation III
  12. A Memory-Efficient Implementation
  13. Main Results
  14. Strongly Convex Case
  15. Convex Case
  16. ...and 21 more sections

Key Result

Lemma 1

Consider any one of the three implementations in Section subsection:impl or the memory-efficient implementation in Section subsection:mem_eff with $p=BR/(nd)$, we have $R/(nd)\leq\sigma \leq 2R/(nd)$ and $\nu=nd\sigma$.

Figures (2)

  • Figure 1: Comparison of different zeroth-order methods on logistic regression problems, the $x$-axis is the number of zeroth-order oracle calls and the $y$-axis is the optimality gap $h(x^{k})- h(x^{\ast})$.
  • Figure 2: Comparison of different zeroth-order methods on the cox regression problem.

Theorems & Definitions (36)

  • Remark 1
  • Lemma 1
  • Theorem 1
  • Corollary 1
  • Theorem 2
  • Corollary 2
  • Definition 1
  • Theorem 3
  • Corollary 3
  • Lemma 2
  • ...and 26 more