Optimistic Critic Reconstruction and Constrained Fine-Tuning for General Offline-to-Online RL

TL;DR

This work addresses the general offline-to-online reinforcement learning (O2O RL) problem, where two key mismatches—evaluation mismatch in value estimation and improvement mismatch in policy updates—impede transferring a policy learned offline to online fine-tuning. The authors introduce a general O2O framework built on three components: policy re-evaluation to obtain optimistic Q-values for the offline critic, value alignment to calibrate the critic with the offline policy, and constrained fine-tuning to mitigate distribution shift during online learning; they instantiate this framework for SAC (O2SAC), TD3 (O2TD3), and PPO (O2PPO). Empirical results on D4RL MuJoCo and AntMaze show that O2O methods achieve stable and efficient performance improvements, often surpassing state-of-the-art baselines and exhibiting strong transferability across diverse offline methods. The proposed approach provides a versatile, general-purpose pathway for leveraging any offline method to any online algorithm, with practical implications for safer and more reliable deployment of offline-trained policies in online settings.

Abstract

Offline-to-online (O2O) reinforcement learning (RL) provides an effective means of leveraging an offline pre-trained policy as initialization to improve performance rapidly with limited online interactions. Recent studies often design fine-tuning strategies for a specific offline RL method and cannot perform general O2O learning from any offline method. To deal with this problem, we disclose that there are evaluation and improvement mismatches between the offline dataset and the online environment, which hinders the direct application of pre-trained policies to online fine-tuning. In this paper, we propose to handle these two mismatches simultaneously, which aims to achieve general O2O learning from any offline method to any online method. Before online fine-tuning, we re-evaluate the pessimistic critic trained on the offline dataset in an optimistic way and then calibrate the misaligned critic with the reliable offline actor to avoid erroneous update. After obtaining an optimistic and and aligned critic, we perform constrained fine-tuning to combat distribution shift during online learning. We show empirically that the proposed method can achieve stable and efficient performance improvement on multiple simulated tasks when compared to the state-of-the-art methods.

Figures (14)

• Figure 1: The results of actors updated with different critics.
• Figure 2: Performance curves on D4RL fu2020d4rl MuJoCo locomotion tasks during online fine-tuning.
• Figure 3: The fine-tuning performance achieved by transferring to three online algorithms from their heterogeneous offline algorithms.
• Figure 4: Performance of our O2PPO and direct PPO from IQL on D4RL fu2020d4rl MuJoCo locomotion tasks during online fine-tuning. The solid lines and shaded regions represent mean and standard deviation.
• Figure 5: Ablation results of our methods, PR=Policy re-evaluation, VA=Value Alignment, CF=Constrained Fine-tuning. For O2PPO, VA means the use of the auxiliary advantage, and CF means the update of the reference policy.

Theorems & Definitions (4)

• Corollary 4.2
• Proposition 4.3
• Proposition 4.4
• Corollary 4.5