Exposing Vulnerabilities in RL: A Novel Stealthy Backdoor Attack through Reward Poisoning

TL;DR

This work reveals a security vulnerability in reinforcement learning by introducing a stealthy backdoor through reward poisoning. It develops a black-box, bi-level optimization framework that perturbs rewards minimally while embedding a target policy that is activated by a trigger. The method demonstrates strong backdoor efficacy across CartPole, Hopper, and Walker2D with minimal normal-performance degradation, highlighting the need for defenses against training-time manipulation. The study also compares against baselines and discusses limitations and future defense directions.

Abstract

Reinforcement learning (RL) has achieved remarkable success across diverse domains, enabling autonomous systems to learn and adapt to dynamic environments by optimizing a reward function. However, this reliance on reward signals creates a significant security vulnerability. In this paper, we study a stealthy backdoor attack that manipulates an agent's policy by poisoning its reward signals. The effectiveness of this attack highlights a critical threat to the integrity of deployed RL systems and calls for urgent defenses against training-time manipulation. We evaluate the attack across classic control and MuJoCo environments. The backdoored agent remains highly stealthy in Hopper and Walker2D, with minimal performance drops of only 2.18 % and 4.59 % under non-triggered scenarios, while achieving strong attack efficacy with up to 82.31% and 71.27% declines under trigger conditions.

Figures (2)

• Figure 1: The proposed attack scheme unfolds across two phases. During the training phase, the attacker intercepts the agent's environmental interaction, and uses the data to update its attack strategy model, i.e., the reward perturbation network ($\Delta$) and the Q-value network ($\bar{Q}$). After adding reward perturbation $\Delta$ to authentic reward data, the poisoned data is then transferred to the agent's replay buffer, guiding it to learn the target backdoor policy. During the deployment phase, the embedded backdoor is activated when the attacker inserts a specific trigger. The stealth of the attack lies in the agent's nominal behavior, which degrades catastrophically only upon activation of a trigger.
• Figure 2: The circled areas indicate where the triggers are inserted. The trigger is introduced by modifying the angle information corresponding to the circled points in the agent's observation.