Seizing Serendipity: Exploiting the Value of Past Success in Off-Policy Actor-Critic

TL;DR

This work identifies a neglected underestimation issue in late-stage off-policy actor-critic RL caused by target-update actions drawn from the current suboptimal policy. It introduces the Blended Exploitation and Exploration (BEE) operator, which blends a Bellman Exploitation component that leverages past high-quality replay-buffer actions with a Bellman Exploration component that preserves optimism, controlled by a parameter $\lambda$. The authors instantiate two practical algorithms, BAC (model-free) and MB-BAC (model-based), showing superior performance and robustness across 50+ continuous-control tasks and real-world quadruped robotics, with improved stability and sample efficiency. Theoretical results establish that the BEE operator is a $\gamma$-contraction and preserves policy-improvement properties, ensuring convergence to favorable fixed points. Overall, the method provides a simple, flexible augmentation to off-policy RL that effectively exploits serendipitous past successes to accelerate learning in both simulated and real-world settings.

Abstract

Learning high-quality $Q$-value functions plays a key role in the success of many modern off-policy deep reinforcement learning (RL) algorithms. Previous works primarily focus on addressing the value overestimation issue, an outcome of adopting function approximators and off-policy learning. Deviating from the common viewpoint, we observe that $Q$-values are often underestimated in the latter stage of the RL training process, potentially hindering policy learning and reducing sample efficiency. We find that such a long-neglected phenomenon is often related to the use of inferior actions from the current policy in Bellman updates as compared to the more optimal action samples in the replay buffer. To address this issue, our insight is to incorporate sufficient exploitation of past successes while maintaining exploration optimism. We propose the Blended Exploitation and Exploration (BEE) operator, a simple yet effective approach that updates $Q$-value using both historical best-performing actions and the current policy. Based on BEE, the resulting practical algorithm BAC outperforms state-of-the-art methods in over 50 continuous control tasks and achieves strong performance in failure-prone scenarios and real-world robot tasks. Benchmark results and videos are available at https://jity16.github.io/BEE/.

Figures (55)

• Figure 1: Example of the underestimation issue. In the DKittyWalkRandomDynamics task, when the current policy-generated action is inferior to the best action in the replay buffer, which usually occurs in the later stage of training, SAC is more prone to underestimation pitfalls than BAC. The gap in $Q$ estimation is evaluated by comparing the SAC $Q$-values and the Monte-Carlo $Q$ estimates.
• Figure 2: Normalized $Q$ estimation value comparison. We plot the normalized $Q$-value estimates of two widely-used off-policy actor-critic algorithms, namely SAC and TD3, across various benchmark suites, spanning locomotion and manipulation tasks. The approximated true value $Q_{MC}$ is obtained by Monte-Carlo $Q$ estimates are derived from trajectories sampled using the current policy. The white vertical line marks the separation of the overestimation and underestimation stages during training. Moreover, we disabled the double-Q technique of SAC and TD3 and observed the underestimation issue still occurs.
• Figure 3: $\Delta(\mu,\pi)$ with an SAC or TD3 agent. Blue bars correspond to positive $\Delta(\mu,\pi)$, indicating an optimal policy derived from the replay buffer would outperform the current policy.
• Figure 4: Illustrative figure on target-update actions. Leveraging the high-quality samples from the replay buffer would produce a more optimistic $Q$-value, thus mitigating value underestimation. Practically, the Bellman evaluation operator, whose target-update actions $a'$ are only sampled from the current policy, tends to underestimate it.
• Figure 5: Comparison of three operators on a toy grid world. The agent's goal is to navigate from the bottom of the maze to the top left. The color of each square shows the learned value, red arrows reveal incorrect actions, and question marks indicate unencountered states. (a) employing a pure exploitation operator may lead to insufficient exploration, causing the agent to miss out on blocks. Conversely, (b) a pure exploration operator $\mathcal{T}_{explore}^\pi$ tends to overestimate the value of less frequently visited areas while underestimating those on the optimal path, resulting in increased sample complexity, (c) Our BEE operator, however, balances between exploitation and exploration, improving convergence and accuracy. And (d) reveals the optimal Q and actions.

Theorems & Definitions (10)

• Definition 4.1
• Proposition 4.2: Policy evaluation
• Proposition 4.3: Policy improvement
• Proposition 4.4: Policy iteration
• Proposition 2.1: Policy evaluation
• proof
• Proposition 2.2: Policy improvement
• proof
• Proposition 2.3: Policy iteration
• proof