Actor-Critics Can Achieve Optimal Sample Efficiency

TL;DR

This work tackles the challenge of learning an $\epsilon$-optimal policy with sample complexity $O(1/\epsilon^2)$ in the presence of general function approximation and necessary exploration. It introduces optimistic actor-critic methods, notably DOUHUA and NORA, that pair a critic targeting $Q^*$ with rare-switching updates and policy resets to achieve sublinear regret and $1/\epsilon^2$ sample efficiency; the main results place the bound at $O\big(dH^5 \log|\mathcal{A}|/\epsilon^2 + dH^4 \log|\mathcal{F}|/\epsilon^2\big)$ with a $\sqrt{T}$ regret when the Bellman eluder dimension $d$ grows slowly. The paper extends the framework to hybrid RL, showing that offline data can yield further gains and presenting NOAH variants that achieve optimism-free $\sqrt{T}$ regret under sufficient offline coverage, with a $1/\epsilon^2$-type guarantee. Numerical experiments in linear and deep settings corroborate the theoretical findings and demonstrate practical viability. Overall, the results significantly advance the understanding of sample-efficient, exploration-aware actor-critic methods for general function approximation and hybrid data regimes, with potential impact on scalable RL in complex environments.

Abstract

Actor-critic algorithms have become a cornerstone in reinforcement learning (RL), leveraging the strengths of both policy-based and value-based methods. Despite recent progress in understanding their statistical efficiency, no existing work has successfully learned an $ε$-optimal policy with a sample complexity of $O(1/ε^2)$ trajectories with general function approximation when strategic exploration is necessary. We address this open problem by introducing a novel actor-critic algorithm that attains a sample-complexity of $O(dH^5 \log|\mathcal{A}|/ε^2 + d H^4 \log|\mathcal{F}|/ ε^2)$ trajectories, and accompanying $\sqrt{T}$ regret when the Bellman eluder dimension $d$ does not increase with $T$ at more than a $\log T$ rate. Here, $\mathcal{F}$ is the critic function class, $\mathcal{A}$ is the action space, and $H$ is the horizon in the finite horizon MDP setting. Our algorithm integrates optimism, off-policy critic estimation targeting the optimal Q-function, and rare-switching policy resets. We extend this to the setting of Hybrid RL, showing that initializing the critic with offline data yields sample efficiency gains compared to purely offline or online RL. Further, utilizing access to offline data, we provide a \textit{non-optimistic} provably efficient actor-critic algorithm that only additionally requires $N_{\text{off}} \geq c_{\text{off}}^*dH^4/ε^2$ in exchange for omitting optimism, where $c_{\text{off}}^*$ is the single-policy concentrability coefficient and $N_{\text{off}}$ is the number of offline samples. This addresses another open problem in the literature. We further provide numerical experiments to support our theoretical findings.

Figures (4)

• Figure 1: Illustration of tracking error in policy optimization, with a rare-switching critic $f^{(t)}$ that targets $\pi^*$. The {blue and pink, pink} area depicts the tracking error of {$\pi^*$ to $\pi^{(t)}$, $\pi^{f^{(t)}}$ to $\pi^{(t)}$}. Both incur $\sqrt{T}$ regret. In contrast, $f^{(t,\pi^{(t)}})$ is a rare-switching critic that targets $\pi^{(t)}$, and so is insufficiently optimistic as $\pi^{(t)}$ changes. The blue, pink, and rust area depicts the tracking error of $\pi^{*}$ to $\pi^{(t_{\text{last}})}$ from insufficient optimism, which yields linear regret.
• Figure 2: Per-episode reward of Algorithms 1 and 2, compared to a rare-switching version of LSVI-UCB (Jin et al., 2019) on a linear MDP tetris task. Here, the condition required for Algorithm 1 to work holds, as we do not clip the Q-function estimates. Algorithm 1 outperforms LSVI-UCB, and Alg. 2 catches up after some time. Results averaged over 30 trials.
• Figure 3: Cumulative regret of Algorithms 1 and 2 vs. LSVI-UCB (Jin et al., 2019) on Tetris. This setting favors Alg. 1, which outperforms LSVI-UCB; Alg. 2 also achieves $\sqrt{T}$ regret. Averaged over 30 trials.
• Figure 4: Hybrid RL experiment on the antmaze-medium-diverse-v2 task. Comparison of Alg. 1H, Alg. 2H, and Cal-QL (Nakamoto et al., 2023) with offline pretraining. Alg. 2H uses approximate optimism via action sampling and outperforms both Cal-QL and Alg. 1H, despite no pessimism. Results suggest hybrid RL enables efficient exploration without pessimism. Evaluation plots show offline pretraining in the first 1000 steps. All plots are exponentially smoothed.

Theorems & Definitions (58)

• Definition 1: Squared Distributional Bellman Eluder dimension jin2021bellmanxiong2023generalframeworksequentialdecisionmaking
• Definition 2: Sequential Extrapolation Coefficient (SEC)
• Theorem 1: Regret Bound for DOUHUA
• proof : Proof sketch:
• Lemma 1: Bound on Covering Number of Policy Class, Modified Lemma B.2 from zhong2023theoreticalanalysisoptimisticproximal
• Definition 3: Closure Under Truncated Sums
• Lemma 2: Policy Class Growth
• Corollary 1: Regret of DOUHUA, The Good Case
• Theorem 2: Regret Bound for NORA
• proof : Proof sketch: