Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Optimal Sample Complexity for Single Time-Scale Actor-Critic with Momentum

Navdeep Kumar, Tehila Dahan, Lior Cohen, Ananyabrata Barua, Giorgia Ramponi, Kfir Yehuda Levy, Shie Mannor

TL;DR

The paper targets the problem of obtaining an $\epsilon$-optimal global policy for infinite-horizon discounted MDPs with finite state-action spaces using a single-timescale actor-critic method. The authors integrate STORM-based variance reduction for the critic with a replay buffer to address nonstationary occupancy measures, enabling a theoretical convergence rate of $O(\epsilon^{-2})$ which matches the information-theoretic lower bound. Key technical advances include deriving interdependent actor/critic/STORM recursions, establishing a Lyapunov function and an extended ODE-domination argument to handle differing step-size schedules, and showing that variance control via momentum plus buffering yields optimal global convergence. The approach is designed to be practical and compatible with standard deep learning architectures, with a transparent memory–trade-off due to retaining the previous Q-network and a buffer. This work closes the gap between upper and lower bounds for single-time-scale AC and offers a principled, variance-aware path to more sample-efficient RL in practice.

Abstract

We establish an optimal sample complexity of $O(ε^{-2})$ for obtaining an $ε$-optimal global policy using a single-timescale actor-critic (AC) algorithm in infinite-horizon discounted Markov decision processes (MDPs) with finite state-action spaces, improving upon the prior state of the art of $O(ε^{-3})$. Our approach applies STORM (STOchastic Recursive Momentum) to reduce variance in the critic updates. However, because samples are drawn from a nonstationary occupancy measure induced by the evolving policy, variance reduction via STORM alone is insufficient. To address this challenge, we maintain a buffer of small fraction of recent samples and uniformly sample from it for each critic update. Importantly, these mechanisms are compatible with existing deep learning architectures and require only minor modifications, without compromising practical applicability.

Optimal Sample Complexity for Single Time-Scale Actor-Critic with Momentum

TL;DR

The paper targets the problem of obtaining an -optimal global policy for infinite-horizon discounted MDPs with finite state-action spaces using a single-timescale actor-critic method. The authors integrate STORM-based variance reduction for the critic with a replay buffer to address nonstationary occupancy measures, enabling a theoretical convergence rate of which matches the information-theoretic lower bound. Key technical advances include deriving interdependent actor/critic/STORM recursions, establishing a Lyapunov function and an extended ODE-domination argument to handle differing step-size schedules, and showing that variance control via momentum plus buffering yields optimal global convergence. The approach is designed to be practical and compatible with standard deep learning architectures, with a transparent memory–trade-off due to retaining the previous Q-network and a buffer. This work closes the gap between upper and lower bounds for single-time-scale AC and offers a principled, variance-aware path to more sample-efficient RL in practice.

Abstract

We establish an optimal sample complexity of for obtaining an -optimal global policy using a single-timescale actor-critic (AC) algorithm in infinite-horizon discounted Markov decision processes (MDPs) with finite state-action spaces, improving upon the prior state of the art of . Our approach applies STORM (STOchastic Recursive Momentum) to reduce variance in the critic updates. However, because samples are drawn from a nonstationary occupancy measure induced by the evolving policy, variance reduction via STORM alone is insufficient. To address this challenge, we maintain a buffer of small fraction of recent samples and uniformly sample from it for each critic update. Importantly, these mechanisms are compatible with existing deep learning architectures and require only minor modifications, without compromising practical applicability.
Paper Structure (26 sections, 24 theorems, 95 equations, 2 figures, 4 tables, 1 algorithm)

This paper contains 26 sections, 24 theorems, 95 equations, 2 figures, 4 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Preliminaries
  3. STORM
  4. Main
  5. Convergence Analysis
  6. Deriving Recursions
  7. GDL Recursion.
  8. Analyzing Recursions
  9. Experiments
  10. Discussion
  11. The Memory–Sample Complexity Trade-off.
  12. Related Work
  13. Exact Policy Gradient (PPG):
  14. Stochastic Policy Gradient Descent
  15. Single Time Scale Actor-critic Algorithm:
  16. ...and 11 more sections

Key Result

Lemma 2.1

(Gradient Domination Lemma, PG_softmax_ConvRates) The sub-optimality is upper bounded by the norm of the gradient as where $C_{PL} = \max_{k}\lVert\frac{d^{\pi^*}}{d^{\pi_{\theta_k}}}\rVert_\infty$ is mismatch coefficient, $c= \min_{k,s}\pi_{\theta_k}(a^*(s)|s)$, and $a^*(s)$ is optimal action in state $s$.

Figures (2)

  • Figure 1: The figure illustrates the key intuition of ODE Domination Lemma, how the ode function $f(k)$ dominates the recursion $y_k$ that is $f(k)\geq y_k$ for all $k$. The recursion is given by $y_{k+1} = y_k - \omega y_k^2$ and the function $\nabla f(x) = -\omega f^2(x)$ with initial condition $y_0=f(0)$. The red dotted lines are the tangent at $x_k$ whose slope is $-\omega f^2(x_k)$ which equals to $-\omega y_k^2$ by construction.
  • Figure 2: Performance our Algorithm \ref{['main:alg:AC']} on a random MDP compared to standard actor-critic without momentum (Algorithm 1 of kumar2025ConvSingTimeAC). MDP parameters: $S=10, A=5, \gamma = 0.9$.

Theorems & Definitions (44)

  • Lemma 2.1
  • Theorem 3.2: Main Result
  • Lemma 4.1
  • proof
  • Lemma 4.2
  • Lemma 4.3
  • Lemma 4.4: Lyapunov Recursion
  • proof
  • Lemma 4.5: ODE Domination Lemma
  • proof
  • ...and 34 more