Improving Policy Exploitation in Online Reinforcement Learning with Instant Retrospect Action

TL;DR

This work tackles slow policy exploitation in online value-based reinforcement learning by introducing Instant Retrospect Action (IRA), which combines Q-representation discrepancy evolution (RDE), greedy action guidance (GAG), and an instant policy update (IPU) mechanism. RDE regularizes the Q-network representations to discriminate neighboring state-action pairs, while GAG imposes explicit, stepwise-constrained policy updates toward top-$k$ neighbor actions, anchored by high-value actions. IPU further accelerates policy exploitation by reducing update delays, and early-stage conservatism helps mitigate overestimation bias. Across eight MuJoCo tasks, IRA consistently outperforms strong baselines (TD3, DDPG, PPO, ALH, PEER, MBPO) and demonstrates robust ablation results, confirming that the combination of representation-guided signals, explicit anchors, and rapid updates enhances both sample efficiency and final performance in online value-based RL.

Abstract

Existing value-based online reinforcement learning (RL) algorithms suffer from slow policy exploitation due to ineffective exploration and delayed policy updates. To address these challenges, we propose an algorithm called Instant Retrospect Action (IRA). Specifically, we propose Q-Representation Discrepancy Evolution (RDE) to facilitate Q-network representation learning, enabling discriminative representations for neighboring state-action pairs. In addition, we adopt an explicit method to policy constraints by enabling Greedy Action Guidance (GAG). This is achieved through backtracking historical actions, which effectively enhances the policy update process. Our proposed method relies on providing the learning algorithm with accurate $k$-nearest-neighbor action value estimates and learning to design a fast-adaptable policy through policy constraints. We further propose the Instant Policy Update (IPU) mechanism, which enhances policy exploitation by systematically increasing the frequency of policy updates. We further discover that the early-stage training conservatism of the IRA method can alleviate the overestimation bias problem in value-based RL. Experimental results show that IRA can significantly improve the learning efficiency and final performance of online RL algorithms on eight MuJoCo continuous control tasks.

Figures (11)

• Figure 1: We introduce auxiliary signals to enhance learning capability and propose two core mechanisms: integrating representation-guided signals into Q-learning and introducing anchor points for policy updates.
• Figure 2: Images for eight MuJoCo environments used in our experiments.
• Figure 3: Learning curves on four MuJoCo continuous control tasks. The shaded region represents half a standard deviation of the average evaluation over 10 trials. Curves are smoothed uniformly for visual clarity.
• Figure 4: Aggregate performance of IRA across eight continuous control tasks using the RLiable analysis framework, reporting Mean, IQM, and Median metrics over the entire empirical evaluation.
• Figure 5: Learning curves obtained with varying $k$ across four MuJoCo continuous control tasks. The shaded region represents half a standard deviation of the average evaluation over 10 trials. Curves are smoothed uniformly for visual clarity.