Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Towards Robust Policy: Enhancing Offline Reinforcement Learning with Adversarial Attacks and Defenses

Thanh Nguyen, Tung M. Luu, Tri Ton, Chang D. Yoo

TL;DR

This work tackles robustness in offline reinforcement learning by introducing a framework that trains policies under adversarial observation perturbations targeting both the actor and critic. It combines four targeted attacks (Random, Critic, Robust Critic, Actor) with two defense regularizers (Critic Defense, Actor Defense) and evaluates them on the D4RL benchmark using standard model-free offline RL methods. The results show substantial vulnerability to perturbations, especially under the Robust Critical Attack, but demonstrate that the proposed defenses can significantly improve policy robustness across datasets, albeit with increased training cost. The approach advances the reliability of offline RL in real-world settings by proactively hardening policies against observation-based attacks.

Abstract

Offline reinforcement learning (RL) addresses the challenge of expensive and high-risk data exploration inherent in RL by pre-training policies on vast amounts of offline data, enabling direct deployment or fine-tuning in real-world environments. However, this training paradigm can compromise policy robustness, leading to degraded performance in practical conditions due to observation perturbations or intentional attacks. While adversarial attacks and defenses have been extensively studied in deep learning, their application in offline RL is limited. This paper proposes a framework to enhance the robustness of offline RL models by leveraging advanced adversarial attacks and defenses. The framework attacks the actor and critic components by perturbing observations during training and using adversarial defenses as regularization to enhance the learned policy. Four attacks and two defenses are introduced and evaluated on the D4RL benchmark. The results show the vulnerability of both the actor and critic to attacks and the effectiveness of the defenses in improving policy robustness. This framework holds promise for enhancing the reliability of offline RL models in practical scenarios.

Towards Robust Policy: Enhancing Offline Reinforcement Learning with Adversarial Attacks and Defenses

TL;DR

This work tackles robustness in offline reinforcement learning by introducing a framework that trains policies under adversarial observation perturbations targeting both the actor and critic. It combines four targeted attacks (Random, Critic, Robust Critic, Actor) with two defense regularizers (Critic Defense, Actor Defense) and evaluates them on the D4RL benchmark using standard model-free offline RL methods. The results show substantial vulnerability to perturbations, especially under the Robust Critical Attack, but demonstrate that the proposed defenses can significantly improve policy robustness across datasets, albeit with increased training cost. The approach advances the reliability of offline RL in real-world settings by proactively hardening policies against observation-based attacks.

Abstract

Offline reinforcement learning (RL) addresses the challenge of expensive and high-risk data exploration inherent in RL by pre-training policies on vast amounts of offline data, enabling direct deployment or fine-tuning in real-world environments. However, this training paradigm can compromise policy robustness, leading to degraded performance in practical conditions due to observation perturbations or intentional attacks. While adversarial attacks and defenses have been extensively studied in deep learning, their application in offline RL is limited. This paper proposes a framework to enhance the robustness of offline RL models by leveraging advanced adversarial attacks and defenses. The framework attacks the actor and critic components by perturbing observations during training and using adversarial defenses as regularization to enhance the learned policy. Four attacks and two defenses are introduced and evaluated on the D4RL benchmark. The results show the vulnerability of both the actor and critic to attacks and the effectiveness of the defenses in improving policy robustness. This framework holds promise for enhancing the reliability of offline RL models in practical scenarios.
Paper Structure (12 sections, 15 equations, 1 figure, 1 table, 1 algorithm)

This paper contains 12 sections, 15 equations, 1 figure, 1 table, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related work
  3. Background
  4. Methodology
  5. The framework for robust offline RL training
  6. Adversarial attack for offline RL
  7. Adversarial defenses for offline reinforcement learning
  8. Experiments
  9. Experiment setup
  10. Experimental Results
  11. Conclusion
  12. Acknowledgement

Figures (1)

  • Figure 1: Aggregated Performance of Defenses with 95% Confidence Intervals (CIs). The aggregated performances are calculated as the average performance over tasks and attacks, specifically on certain types of datasets. The overall performance is further averaged over different datasets.