RAT: Adversarial Attacks on Deep Reinforcement Agents for Targeted Behaviors

TL;DR

RAT addresses robustness evaluation and targeted manipulation of DRL agents by learning a human-aligned intention policy through preference-based RL and guiding an adversary to enforce that target via state perturbations. A weighting function rebalances the replay buffer to focus training on important states, and the method is formalized as a bi-level optimization with provable convergence properties. Empirically, RAT outperforms baselines in manipulating SAC-based DRL agents on Meta-world tasks and can induce human-preferred behaviors in Decision Transformer agents on MuJoCo tasks; it also enables robust-training variants (RAT-ATLA, RAT-WocaR) that improve resilience. The work demonstrates significant implications for safety and robustness, and points to extensions to LLMs and VLA models as future directions.

Abstract

Evaluating deep reinforcement learning (DRL) agents against targeted behavior attacks is critical for assessing their robustness. These attacks aim to manipulate the victim into specific behaviors that align with the attacker's objectives, often bypassing traditional reward-based defenses. Prior methods have primarily focused on reducing cumulative rewards; however, rewards are typically too generic to capture complex safety requirements effectively. As a result, focusing solely on reward reduction can lead to suboptimal attack strategies, particularly in safety-critical scenarios where more precise behavior manipulation is needed. To address these challenges, we propose RAT, a method designed for universal, targeted behavior attacks. RAT trains an intention policy that is explicitly aligned with human preferences, serving as a precise behavioral target for the adversary. Concurrently, an adversary manipulates the victim's policy to follow this target behavior. To enhance the effectiveness of these attacks, RAT dynamically adjusts the state occupancy measure within the replay buffer, allowing for more controlled and effective behavior manipulation. Our empirical results on robotic simulation tasks demonstrate that RAT outperforms existing adversarial attack algorithms in inducing specific behaviors. Additionally, RAT shows promise in improving agent robustness, leading to more resilient policies. We further validate RAT by guiding Decision Transformer agents to adopt behaviors aligned with human preferences in various MuJoCo tasks, demonstrating its effectiveness across diverse tasks.

Figures (10)

• Figure 1: An example illustrating the distinction between our approach and generic attacks.
• Figure 2: Diagram of PbRL. The reward model $\widehat{r}_\psi$ is trained to align with human intention, providing estimations of rewards for policy learning. The policy is optimized by using transitions relabeled by the up-to-date reward model.
• Figure 3: Overview of RAT. During training, it learns the intention policy $\pi_\theta$ and the reward model $\widehat{r}_\psi$, following the principles of PbRL. Simultaneously, it trains an adversary $\pi_\alpha$ and a weighting function $h_\omega$ within a bi-level optimization framework. In the inner-level, the adversary is optimized such that the perturbed policy aligns with the intention policy. A validation loss $J_{\pi}$ is introduced, serving as a metric to assess the adversary's performance. In the outer-level, the weighting function is updated to improve the performance of the adversary by minimizing the outer loss $J_{\pi}$.
• Figure 4: Human desired behaviors behaved by the Decision Transformer under the attack of RAT.
• Figure 5: Effects of the Weighting Function. (a) Trajectory weights generated by the weighting function from various policies are visualized with t-SNE. (b) A visualization of the weights of trajectories of different qualities by five different policies.

Theorems & Definitions (11)

• Lemma 3.1
• Theorem 4.1
• Theorem 4.2
• Lemma C.1
• Lemma D.1
• proof
• Theorem D.2
• proof
• Lemma D.3
• Theorem D.4