TooBadRL: Trigger Optimization to Boost Effectiveness of Backdoor Attacks on Deep Reinforcement Learning

TL;DR

TooBadRL addresses the security of deep reinforcement learning by introducing a systematic trigger-optimization framework for backdoor attacks. It jointly optimizes injection timing, trigger dimension via SHAP-based attribution, and manipulation magnitude through gradient-based optimization, coupled with adaptive training-time strategies and backdoor implantation techniques. Empirical results across three DRL algorithms and nine environments show robust improvements in attack effectiveness (high ASR) while preserving normal task performance (high NTP), achieving superior BUS relative to baselines and exhibiting resilience against Neural Cleanse and RL-Sanitization. The work highlights the practical risks of systematically designed backdoors in DRL and provides open-source code to spur further defense research and reproducibility.

Abstract

Deep reinforcement learning (DRL) has achieved remarkable success in a wide range of sequential decision-making applications, including robotics, healthcare, smart grids, and finance. Recent studies reveal that adversaries can implant backdoors into DRL agents during the training phase. These backdoors can later be activated by specific triggers during deployment, compelling the agent to execute targeted actions and potentially leading to severe consequences, such as drone crashes or vehicle collisions. However, existing backdoor attacks utilize simplistic and heuristic trigger configurations, overlooking the critical impact of trigger design on attack effectiveness. To address this gap, we introduce TooBadRL, the first framework to systematically optimize DRL backdoor triggers across three critical aspects: injection timing, trigger dimension, and manipulation magnitude. Specifically, we first introduce a performance-aware adaptive freezing mechanism to determine the injection timing during training. Then, we formulate trigger selection as an influence attribution problem and apply Shapley value analysis to identify the most influential trigger dimension for injection. Furthermore, we propose an adversarial input synthesis method to optimize the manipulation magnitude under environmental constraints. Extensive evaluations on three DRL algorithms and nine benchmark tasks demonstrate that TooBadRL outperforms five baseline methods in terms of attack success rate while only slightly affecting normal task performance. We further evaluate potential defense strategies from detection and mitigation perspectives. We open-source our code to facilitate reproducibility and further research.

Figures (7)

• Figure 1: Impact of trigger configuration on attack performance. Each subfigure illustrates how varying one critical aspect affects both ASR and NTP.
• Figure 2: The framework of TooBadRL. TooBadRL operates through two stages: (1) Trigger Optimization systematically optimizes injection timing, trigger dimension, and manipulation magnitude; (2) Backdoor Implantation embeds the backdoor through trigger injection, dynamic attack frequency adaptation, action manipulation, and reward modification.
• Figure 3: Evolution of NTP and ASR when trigger injection begins from the start of training (without freezing period), contrasted with the NTP of a clean agent.
• Figure 4: Visualization of trigger dimensions in DRL state spaces. Each state dimension corresponds to a specific physical property.
• Figure 5: Visualization of manipulation magnitude in Pendulum. Different values correspond to distinct physical states. (a) shows the x-coordinate dimension. (b) represents the maximum value of 1 (upright position), (c) represents the midpoint of 0 (horizontal position), and (d) represents the minimum value of -1 (downward position).