Adversarial Fine-tuning in Offline-to-Online Reinforcement Learning for Robust Robot Control

TL;DR

The paper tackles robustness of offline reinforcement learning to action-space perturbations in robot control. It proposes an offline-to-online framework with adversarial fine-tuning, where an offline-pretrained policy is refined online in perturbed environments, and uses a performance-aware curriculum that modulates perturbation probability via an exponential moving average. Perturbations are precomputed with differential evolution to avoid inner-loop optimization, and TD3+BC is used for offline pretraining with standard TD3 updates during finetuning. Experiments on Hopper-v2, HalfCheetah-v2, and Ant-v2 show that adversarial fine-tuning improves robustness under perturbations and accelerates convergence relative to offline-only and fully online baselines, with the adaptive curriculum mitigating nominal-performance degradation observed in fixed schedules. Overall, the approach provides a practical bridge between offline efficiency and online adaptability for robust legged locomotion control, with potential extensions to theory and broader perturbation regimes.

Abstract

Offline reinforcement learning enables sample-efficient policy acquisition without risky online interaction, yet policies trained on static datasets remain brittle under action-space perturbations such as actuator faults. This study introduces an offline-to-online framework that trains policies on clean data and then performs adversarial fine-tuning, where perturbations are injected into executed actions to induce compensatory behavior and improve resilience. A performance-aware curriculum further adjusts the perturbation probability during training via an exponential-moving-average signal, balancing robustness and stability throughout the learning process. Experiments on continuous-control locomotion tasks demonstrate that the proposed method consistently improves robustness over offline-only baselines and converges faster than training from scratch. Matching the fine-tuning and evaluation conditions yields the strongest robustness to action-space perturbations, while the adaptive curriculum strategy mitigates the degradation of nominal performance observed with the linear curriculum strategy. Overall, the results show that adversarial fine-tuning enables adaptive and robust control under uncertain environments, bridging the gap between offline efficiency and online adaptability.

Figures (8)

• Figure 1: Incompatibility between conservative offline RL and action-space robustness. Left: A conservatively trained policy matches the dataset (dashed line) under normal conditions (blue) but fails under perturbations (red). Right: Robustness requires compensatory actions that maintain consistent distributions across both conditions (blue and red overlap).
• Figure 2: Overview of the proposed framework with three stages: (1) Offline Pretraining, where a policy $\pi_\text{off}$ is trained on a clean dataset; (2) Adversarial Perturbation Generation, where perturbations $\{\delta_a\}$ are precomputed with $\pi_\text{off}$ before fine-tuning; and (3) Adversarial Fine-Tuning, where $\pi_\text{off}$ is refined in perturbed environments using these perturbations and replay buffer updates.
• Figure 3: Legged robot environments: Hopper-v2 (left), HalfCheetah-v2 (center), Ant-v2 (right).
• Figure 4: Performance curves show that fine-tuning accelerates convergence and improves robustness. Matching training and evaluation conditions (e.g., FT-A for adversarial evaluation, red curves in right column) yields fastest convergence. Offline baselines (dashed) excel in normal conditions but fail under perturbations. Rows: Hopper-v2, HalfCheetah-v2, Ant-v2. Columns: evaluation under normal, random, and adversarial perturbations. Shaded regions show standard deviation.
• Figure 5: The trajectories of the perturbation probability $q$ illustrate the key behavioral difference between the two curricula. The adaptive strategy ($q_\text{ada}$, red) adjusts the growth of $q$ in response to policy performance, preventing the relentless increase seen in the linear strategy ($q_\text{lin}$, blue).