Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Robust Deep Reinforcement Learning with Adaptive Adversarial Perturbations in Action Space

Qianmei Liu, Yufei Kuang, Jie Wang

TL;DR

This work tackles the challenge of achieving both high nominal performance and robustness in deep reinforcement learning under model mismatch. It introduces Adaptive Adversarial Perturbation (A2P), which dynamically adjusts perturbation strength in the action space through an adaptive coefficient driven by training dynamics, eliminating the need for pre-access to simulators. The proposed A2P-MDP framework and its SAC-based instantiation (A2P-SAC) demonstrate that adaptive perturbations improve training stability and policy robustness across MuJoCo tasks with varying masses and frictions, outperforming fixed-perturbation baselines. The results highlight the practical potential of adaptive, action-space adversarial training for real-world DRL deployments where simulator access is limited or unavailable. The approach is supported by theoretical guarantees of contraction for the adaptive Bellman operator and a clear strategy for updating perturbation strength based on observed action-distance dynamics.

Abstract

Deep reinforcement learning (DRL) algorithms can suffer from modeling errors between the simulation and the real world. Many studies use adversarial learning to generate perturbation during training process to model the discrepancy and improve the robustness of DRL. However, most of these approaches use a fixed parameter to control the intensity of the adversarial perturbation, which can lead to a trade-off between average performance and robustness. In fact, finding the optimal parameter of the perturbation is challenging, as excessive perturbations may destabilize training and compromise agent performance, while insufficient perturbations may not impart enough information to enhance robustness. To keep the training stable while improving robustness, we propose a simple but effective method, namely, Adaptive Adversarial Perturbation (A2P), which can dynamically select appropriate adversarial perturbations for each sample. Specifically, we propose an adaptive adversarial coefficient framework to adjust the effect of the adversarial perturbation during training. By designing a metric for the current intensity of the perturbation, our method can calculate the suitable perturbation levels based on the current relative performance. The appealing feature of our method is that it is simple to deploy in real-world applications and does not require accessing the simulator in advance. The experiments in MuJoCo show that our method can improve the training stability and learn a robust policy when migrated to different test environments. The code is available at https://github.com/Lqm00/A2P-SAC.

Robust Deep Reinforcement Learning with Adaptive Adversarial Perturbations in Action Space

TL;DR

This work tackles the challenge of achieving both high nominal performance and robustness in deep reinforcement learning under model mismatch. It introduces Adaptive Adversarial Perturbation (A2P), which dynamically adjusts perturbation strength in the action space through an adaptive coefficient driven by training dynamics, eliminating the need for pre-access to simulators. The proposed A2P-MDP framework and its SAC-based instantiation (A2P-SAC) demonstrate that adaptive perturbations improve training stability and policy robustness across MuJoCo tasks with varying masses and frictions, outperforming fixed-perturbation baselines. The results highlight the practical potential of adaptive, action-space adversarial training for real-world DRL deployments where simulator access is limited or unavailable. The approach is supported by theoretical guarantees of contraction for the adaptive Bellman operator and a clear strategy for updating perturbation strength based on observed action-distance dynamics.

Abstract

Deep reinforcement learning (DRL) algorithms can suffer from modeling errors between the simulation and the real world. Many studies use adversarial learning to generate perturbation during training process to model the discrepancy and improve the robustness of DRL. However, most of these approaches use a fixed parameter to control the intensity of the adversarial perturbation, which can lead to a trade-off between average performance and robustness. In fact, finding the optimal parameter of the perturbation is challenging, as excessive perturbations may destabilize training and compromise agent performance, while insufficient perturbations may not impart enough information to enhance robustness. To keep the training stable while improving robustness, we propose a simple but effective method, namely, Adaptive Adversarial Perturbation (A2P), which can dynamically select appropriate adversarial perturbations for each sample. Specifically, we propose an adaptive adversarial coefficient framework to adjust the effect of the adversarial perturbation during training. By designing a metric for the current intensity of the perturbation, our method can calculate the suitable perturbation levels based on the current relative performance. The appealing feature of our method is that it is simple to deploy in real-world applications and does not require accessing the simulator in advance. The experiments in MuJoCo show that our method can improve the training stability and learn a robust policy when migrated to different test environments. The code is available at https://github.com/Lqm00/A2P-SAC.
Paper Structure (20 sections, 2 theorems, 20 equations, 6 figures, 1 table, 1 algorithm)

This paper contains 20 sections, 2 theorems, 20 equations, 6 figures, 1 table, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Robust Reinforcement Learning (RRL)
  4. Adversarial Training (AT)
  5. Preliminaries
  6. Markov Decision Process (MDP)
  7. Zero-sum Game
  8. Robustness and Average Performance
  9. Method
  10. Motivation
  11. Adaptive Adversarial Perturbation Markov Decision Process (A2P-MDP)
  12. Adaptive Adversarial Perturbation in Action Space
  13. Theoretical Analysis
  14. Adaptive Perturbation for Adversarial Training : A case Study on SAC
  15. Experiments
  16. ...and 5 more sections

Key Result

Theorem 1

For any $\epsilon\in[0,1]$ and any fixed policy, the adaptive adversarial Bellman operator $\mathbf{B}^{adpt}_{\epsilon-\mathcal{A}}$ in Equation eq8 a contraction that converges to $Q^{adpt}_{\epsilon-\mathcal{A}}$ in Equation eq7.

Figures (6)

  • Figure 1: (a) Comparison of performance in source environments with different perturbation methods. (b) Comparison of the relative performance, which compares with mass = 1.0, with different perturbation methods in testing environments. (c) Effects of range-specific perturbations on model performance in the source environment.
  • Figure 2: The general architecture of the A2P method
  • Figure 3: Comparison of SAC, NR-SAC, and A2P-SAC in the target environment with perturbed mass and friction. For each task we randomly choose five seeds for training and save the last 10 episodes of each random seed. For each seed we randomly choose four policies for testing, then calculate the average performance. The results show that A2P-SAC can learn a more robust policy.
  • Figure 4: Comparison of training curves for different algorithms. For each algorithm in each task we train 1000k steps. The results show that our algorithm can approximate the performance of SAC in most tasks and is more stable than NR-SAC.
  • Figure 5: Comparing the robustness of SAC, NR-SAC and A2P-SAC with one perturbed parameter(mass or friction) in target environment. The higher the reward, the better the performance. If the model performs well in multiple test environments, it means that the model has strong robustness. The results show that A2P method learns a more robust policy than others.
  • ...and 1 more figures

Theorems & Definitions (2)

  • Theorem 1: Bellman Contraction.
  • Theorem 2: Policy Improvement.