UNIDOOR: A Universal Framework for Action-Level Backdoor Attacks in Deep Reinforcement Learning

TL;DR

This work addresses the universality gap in action-level backdoor attacks for DRL by reframing backdoor injection as a multi-task learning problem shared by benign and backdoor tasks. It introduces UNIDOOR, a four-module framework consisting of Performance Monitoring, Initial Freezing, Transition Poisoning, and Adaptive Exploration, which adaptively tunes backdoor rewards based on monitored performance indicators. The approach demonstrates broad effectiveness across 11 tasks, 53 backdoor designs, and 3 DRL algorithms, with strong CP scores and visible stealth in state distributions and neuron activations. The findings highlight both the security risk posture of DRL systems and avenues for defense, while also outlining limitations and future research directions, including extensions to multi-backdoor, multi-agent, and offline RL settings.

Abstract

Deep reinforcement learning (DRL) is widely applied to safety-critical decision-making scenarios. However, DRL is vulnerable to backdoor attacks, especially action-level backdoors, which pose significant threats through precise manipulation and flexible activation, risking outcomes like vehicle collisions or drone crashes. The key distinction of action-level backdoors lies in the utilization of the backdoor reward function to associate triggers with target actions. Nevertheless, existing studies typically rely on backdoor reward functions with fixed values or conditional flipping, which lack universality across diverse DRL tasks and backdoor designs, resulting in fluctuations or even failure in practice. This paper proposes the first universal action-level backdoor attack framework, called UNIDOOR, which enables adaptive exploration of backdoor reward functions through performance monitoring, eliminating the reliance on expert knowledge and grid search. We highlight that action tampering serves as a crucial component of action-level backdoor attacks in continuous action scenarios, as it addresses attack failures caused by low-frequency target actions. Extensive evaluations demonstrate that UNIDOOR significantly enhances the attack performance of action-level backdoors, showcasing its universality across diverse attack scenarios, including single/multiple agents, single/multiple backdoors, discrete/continuous action spaces, and sparse/dense reward signals. Furthermore, visualization results encompassing state distribution, neuron activation, and animations demonstrate the stealthiness of UNIDOOR. The source code of UNIDOOR can be found at https://github.com/maoubo/UNIDOOR.

Figures (12)

• Figure 1: The impact of backdoor reward and task variations on the attack performance of action-level backdoors, where color intensity reflects the harmonic mean of the victim agent's performance on both benign and backdoor tasks.
• Figure 2: The schematic diagram of DRL.
• Figure 3: The framework of Unidoor. ① is the monitored BTP, ② is the lift status of Initial Freezing, ③ is the latest backdoor reward, ④ is the monitored BTP and ASR, and ⑤ is the updated backdoor reward.
• Figure 4: The impact trend of backdoor rewards on the performance of benign and backdoor tasks.
• Figure 5: In the expansion and contraction phases, the conservative value ranges of BTP and ASR serve as the criteria for initiating modifications to the backdoor reward space.