ENOTO: Improving Offline-to-Online Reinforcement Learning with Q-Ensembles

TL;DR

The paper tackles the instability and slow progress often seen when moving from offline RL to online fine-tuning. It introduces ENOTO, an ensemble-based framework that uses $N$ Q-networks and ensemble-target strategies (e.g., WeightedMinPair) along with SUNRISE exploration to convert pessimistic offline training into efficient online refinement. By instantiating ENOTO on offline RL baselines like CQL and LAPO, the authors demonstrate improved training stability, faster online learning, and superior final performance across MuJoCo locomotion and Antmaze navigation benchmarks. This approach offers a practical, plug-in solution for deploying offline-trained policies with rapid, reliable online adaptation in real-world robotic and navigation tasks.

Abstract

Offline reinforcement learning (RL) is a learning paradigm where an agent learns from a fixed dataset of experience. However, learning solely from a static dataset can limit the performance due to the lack of exploration. To overcome it, offline-to-online RL combines offline pre-training with online fine-tuning, which enables the agent to further refine its policy by interacting with the environment in real-time. Despite its benefits, existing offline-to-online RL methods suffer from performance degradation and slow improvement during the online phase. To tackle these challenges, we propose a novel framework called ENsemble-based Offline-To-Online (ENOTO) RL. By increasing the number of Q-networks, we seamlessly bridge offline pre-training and online fine-tuning without degrading performance. Moreover, to expedite online performance enhancement, we appropriately loosen the pessimism of Q-value estimation and incorporate ensemble-based exploration mechanisms into our framework. Experimental results demonstrate that ENOTO can substantially improve the training stability, learning efficiency, and final performance of existing offline RL methods during online fine-tuning on a range of locomotion and navigation tasks, significantly outperforming existing offline-to-online RL methods.

Figures (15)

• Figure 1: (a) Normalized return curves of some motivated examples while performing online fine-tuning with offline policy trained on Walker2d-medium-expert-v2 dataset. (b) Comparison of the average Q-values of SAC and SAC-N. (c) Histograms of the distances between the actions from each method (CQL, SAC-N, and a random policy) and the actions from the dataset.
• Figure 2: Aggregated learning curves of different offline-to-online RL approaches on all considered MuJoCo datasets.
• Figure 3: Aggregated learning curves of OnlineRL-N using different Q-target computation methods on all considered MuJoCo datasets.
• Figure 4: Aggregated learning curves of OnlineRL-N + WeightedMinPair using different exploration methods on all considered MuJoCo datasets.
• Figure 5: Online learning curves of different methods across five seeds on MuJoCo locomotion tasks. The solid lines and shaded regions represent mean and standard deviation, respectively.