Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Reward-Preserving Attacks For Robust Reinforcement Learning

Lucas Schott, Elies Gherbi, Hatem Hajri, Sylvain Lamprier

TL;DR

This work proposes $\alpha$-reward-preserving attacks, which adapt the strength of the adversary so that an $\alpha$ fraction of the nominal-to-worst-case return gap remains achievable at each state.

Abstract

Adversarial robustness in RL is difficult because perturbations affect entire trajectories: strong attacks can break learning, while weak attacks yield little robustness, and the appropriate strength varies by state. We propose $α$-reward-preserving attacks, which adapt the strength of the adversary so that an $α$ fraction of the nominal-to-worst-case return gap remains achievable at each state. In deep RL, we use a gradient-based attack direction and learn a state-dependent magnitude $η\le η_{\mathcal B}$ selected via a critic $Q^π_α((s,a),η)$ trained off-policy over diverse radii. This adaptive tuning calibrates attack strength and, with intermediate $α$, improves robustness across radii while preserving nominal performance, outperforming fixed- and random-radius baselines.

Reward-Preserving Attacks For Robust Reinforcement Learning

TL;DR

This work proposes -reward-preserving attacks, which adapt the strength of the adversary so that an fraction of the nominal-to-worst-case return gap remains achievable at each state.

Abstract

Adversarial robustness in RL is difficult because perturbations affect entire trajectories: strong attacks can break learning, while weak attacks yield little robustness, and the appropriate strength varies by state. We propose -reward-preserving attacks, which adapt the strength of the adversary so that an fraction of the nominal-to-worst-case return gap remains achievable at each state. In deep RL, we use a gradient-based attack direction and learn a state-dependent magnitude selected via a critic trained off-policy over diverse radii. This adaptive tuning calibrates attack strength and, with intermediate , improves robustness across radii while preserving nominal performance, outperforming fixed- and random-radius baselines.
Paper Structure (29 sections, 1 theorem, 45 equations, 6 figures, 4 algorithms)

This paper contains 29 sections, 1 theorem, 45 equations, 6 figures, 4 algorithms.

Table of Contents

  1. Introduction
  2. Toward Robust RL with Reward-Preserving Attacks
  3. Robust RL through Adversarial Training
  4. Reward-Preserving Attacks
  5. Magnitude–Direction Decomposition of Perturbations
  6. Approximated Reward-Preserving Attacks for Robust Deep RL
  7. Reference Policies
  8. Approximation of Q-values for the full range of magnitudes in $[0,\eta_{\cal B}]$
  9. Off-policy Updates for non-Stationary State-Action Occupancy
  10. Experiments
  11. Experimental setup
  12. Reward-preserving attacks via a learned $Q^{\tilde{\pi}}_\alpha((s,a),\eta)$ and reward preservation target $\alpha$
  13. Adversarial fine-tuning with our reward-preserving $\alpha$-attacks
  14. Nominal performance during training ($\eta=0$).
  15. Robust performance during training ($\eta=0.2$).
  16. ...and 14 more sections

Key Result

Proposition 1.1

$\mathcal{T}$ is a $\gamma$-contraction in the supremum norm $\|\cdot\|_\infty$, hence it admits a unique fixed point $Q^{*,\Omega^{\xi^*}}$, and iterating converges to $Q^{*,\Omega^{\xi^*}}$ with

Figures (6)

  • Figure 1: Comparison of value functions and induced optimal trajectories on a deterministic GridWorld environment for: (left) classical Value Iteration, (middle) Robust Value Iteration (RVI), and (right) our $\alpha$-reward-preserving extension of RVI ($\alpha=0.3$). The environment contains a single positively rewarded goal state located in the top-left corner, while black cells correspond to terminal states with reward $-1$. A discount factor $\gamma = 0.999$ is used. Robustness is enforced through an uncertainty set $\mathcal{B}$ over transition kernels, where the ambiguity radius $\eta_{\mathcal{B}}$ is computed via a Sinkhorn-regularized $W_2$ transportation cost between next-state distributions, with ground costs defined as Euclidean distances between successor states. Implementation details are provided in Appendix \ref{['sec:gridworld']}.
  • Figure 2: Calibration of reward-preserving $\alpha$-attacks on the pretrained agent using the learned $Q^{\tilde{\pi}}_\alpha((s,a),\eta)$. As $\alpha$ decreases, the attack chooses larger radii on average and reduces return more; as $\alpha \to 1$, perturbations become smaller and return approaches nominal performance.
  • Figure 3: Reward-preserving adversarial fine-tuning: evaluation return at $\eta=0$ over training, for several preservation targets $\alpha$. All $\alpha$-trained agents retain strong nominal performance.
  • Figure 4: Reward-preserving adversarial fine-tuning: evaluation return at $\eta=0.2$ over training, for several preservation targets $\alpha$. Intermediate $\alpha$ values yield the strongest robustness under this perturbation level.
  • Figure 5: Robustness profiles of $\alpha$-trained agents under varying evaluation $\eta$: Agents trained with intermediate $\alpha$ are robust over a broader range of radii than fixed-$\eta$ trained agents.
  • ...and 1 more figures

Theorems & Definitions (5)

  • Definition 2.1: Reward-Preserving Attack
  • Proposition 1.1
  • proof
  • proof
  • proof