Fine-Tuning without Performance Degradation

TL;DR

Fine-tuning policies learned offline often exhibits non-monotonic performance, with early degradation that hinders practical deployment. The authors analyze the degradation mechanisms and introduce Automatic Jump Start (AJS), an adaptive method that combines the stability of offline learning (InAC) with the rapid improvements of online actors (SAC) by using off-policy evaluation (FQE) to adjust the exploration horizon $h$ automatically. AJS avoids manual hyperparameter tuning, maintains a reliable balance between safety and improvement, and demonstrates faster fine-tuning with reduced degradation on D4RL benchmarks compared to existing approaches. The work advances practical offline-to-online RL by providing a parameter-light, data-driven strategy for controlled exploration that is better suited for real-world applications where deployment-time tuning is limited. Overall, AJS offers a robust path toward safer, more sample-efficient online adaptation of offline-trained policies.

Abstract

Fine-tuning policies learned offline remains a major challenge in application domains. Monotonic performance improvement during \emph{fine-tuning} is often challenging, as agents typically experience performance degradation at the early fine-tuning stage. The community has identified multiple difficulties in fine-tuning a learned network online, however, the majority of progress has focused on improving learning efficiency during fine-tuning. In practice, this comes at a serious cost during fine-tuning: initially, agent performance degrades as the agent explores and effectively overrides the policy learned offline. We show across a range of settings, many offline-to-online algorithms exhibit either (1) performance degradation or (2) slow learning (sometimes effectively no improvement) during fine-tuning. We introduce a new fine-tuning algorithm, based on an algorithm called Jump Start, that gradually allows more exploration based on online estimates of performance. Empirically, this approach achieves fast fine-tuning and significantly reduces performance degradations compared with existing algorithms designed to do the same.

Figures (23)

• Figure 1: All algorithms tested could not prevent performance degradation when fine-tuning a near-optimal policy. The x-axis represents the timestep ($\times 10^4$) during fine-tuning, while the y-axis is the normalized return averaged over 15 runs. The shaded area indicates the 95% bootstrap confidence interval. SAC fine-tuning a the offline InAC policy was included as a baseline. SAC+Ensemble CQL used 10 networks.
• Figure 2: Performance degredation of SAC is related to the quality of the policy used to generate the data for offline training. In all cases, fine-tuned SAC eventually outperforms a the simple baseline of using SAC without offline training (labelled Scratch). The y-axis reports normalized performance averaged over 15 runs, while the x-axis denotes the time step $(\times 10^4)$ during fine-tuning. The shaded area indicates the 95% bootstrap confidence interval. The horizontal dashed line indicates the performance at initialization for each dataset (color matched).
• Figure 3: The effect of overestimation in states that are not covered by the dataset. Each subplot visualizes the state-action pairs in the environment at different time steps. The subtitle specifies whether the agent exploited at state $s_0$. Action values are represented by color intensity, with deeper colors indicating higher estimates. Yellow arrows refer to the path taken by the agent in that episode.
• Figure 4: After 10,000 updates, SAC shifted toward selecting actions with higher initial value estimates. Each scatter represents a randomly sampled state from the offline dataset used for policy learning, with coordinates computed via PCA. The color of each scatter indicates the value difference between the updated policy and the initialized policy, as measured by the initial critic. A more intense red indicates a larger difference, meaning fine-tuning shifts the policy toward actions with higher initial value estimates. We set the color bar in the same environment to the same value range for clear visualization and shorten HalfCheetah to HC.
• Figure 5: InAC demonstrated more stable performance when the policy initialization was near-optimal, but underperformed SAC when the initialization was suboptimal. Each row presents learning curves across different environments, while each column corresponds to a different policy initialization, trained on the Expert, Medium-Expert, and Medium datasets, respectively. The x-axis is the time step $(\times 10^4)$, and the y-axis is the normalized return. The shaded area refers to the $95\%$ confidence interval.