Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Gradient-Free Approaches is a Key to an Efficient Interaction with Markovian Stochasticity

Boris Prokhorov, Semyon Chebykin, Alexander Gasnikov, Aleksandr Beznosikov

TL;DR

The paper studies stochastic optimization with Markovian noise when only a zero-order oracle is available. It introduces a gradient-free, accelerated SGD framework that uses a randomized multilevel Monte Carlo batching scheme and smoothing to achieve convergence rates that remain favorable even under Markovian correlations, notably when the mixing time $τ$ satisfies $τ < d$. The authors establish tight upper bounds for both one-point and two-point feedback in smooth and non-smooth settings, and match them with information-theoretic lower bounds, demonstrating optimality up to logarithmic factors. A key takeaway is that zero-order methods, when paired with batching and smoothing, can efficiently interact with Markovian stochasticity, offering substantial practical efficiency gains over gradient-based methods in this setting. The results have implications for reinforcement learning and online optimization where gradient information is unavailable or expensive to obtain.

Abstract

This paper deals with stochastic optimization problems involving Markovian noise with a zero-order oracle. We present and analyze a novel derivative-free method for solving such problems in strongly convex smooth and non-smooth settings with both one-point and two-point feedback oracles. Using a randomized batching scheme, we show that when mixing time $τ$ of the underlying noise sequence is less than the dimension of the problem $d$, the convergence estimates of our method do not depend on $τ$. This observation provides an efficient way to interact with Markovian stochasticity: instead of invoking the expensive first-order oracle, one should use the zero-order oracle. Finally, we complement our upper bounds with the corresponding lower bounds. This confirms the optimality of our results.

Gradient-Free Approaches is a Key to an Efficient Interaction with Markovian Stochasticity

TL;DR

The paper studies stochastic optimization with Markovian noise when only a zero-order oracle is available. It introduces a gradient-free, accelerated SGD framework that uses a randomized multilevel Monte Carlo batching scheme and smoothing to achieve convergence rates that remain favorable even under Markovian correlations, notably when the mixing time satisfies . The authors establish tight upper bounds for both one-point and two-point feedback in smooth and non-smooth settings, and match them with information-theoretic lower bounds, demonstrating optimality up to logarithmic factors. A key takeaway is that zero-order methods, when paired with batching and smoothing, can efficiently interact with Markovian stochasticity, offering substantial practical efficiency gains over gradient-based methods in this setting. The results have implications for reinforcement learning and online optimization where gradient information is unavailable or expensive to obtain.

Abstract

This paper deals with stochastic optimization problems involving Markovian noise with a zero-order oracle. We present and analyze a novel derivative-free method for solving such problems in strongly convex smooth and non-smooth settings with both one-point and two-point feedback oracles. Using a randomized batching scheme, we show that when mixing time of the underlying noise sequence is less than the dimension of the problem , the convergence estimates of our method do not depend on . This observation provides an efficient way to interact with Markovian stochasticity: instead of invoking the expensive first-order oracle, one should use the zero-order oracle. Finally, we complement our upper bounds with the corresponding lower bounds. This confirms the optimality of our results.
Paper Structure (30 sections, 32 theorems, 193 equations, 2 figures, 4 tables, 1 algorithm)

This paper contains 30 sections, 32 theorems, 193 equations, 2 figures, 4 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related works
  3. Our contribution
  4. Main results
  5. Batching technique
  6. Algorithm
  7. Lower bounds
  8. Experiments
  9. Appendix overview
  10. Additional results
  11. Adversarial noise
  12. Non-smooth
  13. Oracle complexity bound
  14. Notations and definitions.
  15. Proofs of one-point results
  16. ...and 15 more sections

Key Result

Lemma 1

Let as:mixing_timeas:noise(as:noise_two) hold. Then for any $n \geq 1$ and $x \in \mathbb{R}^{d}$ and any initial distribution $\xi$ on $(\mathsf{Z},\mathcal{Z})$, we have

Figures (2)

  • Figure 1: Optimization error $\varepsilon = \|x^N - x^*\|^2$ after $N = 10^3$ iterations. Starting point error $\norm{x_0 - x^*}^2$ = $10^{-2}$. Stepsize $\gamma = 10^{-3}$, $t = 10^{-5}$. The results are averaged over $10^4$ runs.
  • Figure :

Theorems & Definitions (49)

  • Lemma 1
  • Lemma 2: for one-point
  • theorem 1
  • theorem \ref{th:acc}$'$
  • theorem \ref{th:acc}$'$
  • theorem \ref{th:acc}$'$
  • theorem \ref{th:acc_ns}$'$
  • Lemma 3: Extended version of \ref{['lem:tech_markov']}
  • proof
  • Lemma 4
  • ...and 39 more