Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Fox in the Henhouse: Supply-Chain Backdoor Attacks Against Reinforcement Learning

Shijie Liu, Andrew C. Cullen, Paul Montague, Sarah Erfani, Benjamin I. P. Rubinstein

TL;DR

This work shows that reinforcement learning backdoors can be realistically embedded through supply-chain vulnerabilities, using externally sourced agents that only perform legitimate actions. The SCAB framework trains an attacker with a backdoor-rewarding policy and a detector, then injects trigger sequences during victim training to induce backdoor actions and degrade performance, without accessing private training data. Empirical results across multiple games and architectures demonstrate trigger success above $90\%$ with as little as $3\%$ training exposure, and an $\geq80\%$ drop in average episodic return, matching prior attacks under far stronger access assumptions. The study underscores the practical risk of untrusted model provisioning in RL and motivates the development of robust defenses and supply-chain safeguards for RL systems.

Abstract

The current state-of-the-art backdoor attacks against Reinforcement Learning (RL) rely upon unrealistically permissive access models, that assume the attacker can read (or even write) the victim's policy parameters, observations, or rewards. In this work, we question whether such a strong assumption is required to launch backdoor attacks against RL. To answer this question, we propose the \underline{S}upply-\underline{C}h\underline{a}in \underline{B}ackdoor (SCAB) attack, which targets a common RL workflow: training agents using external agents that are provided separately or embedded within the environment. In contrast to prior works, our attack only relies on legitimate interactions of the RL agent with the supplied agents. Despite this limited access model, by poisoning a mere $3\%$ of training experiences, our attack can successfully activate over $90\%$ of triggered actions, reducing the average episodic return by $80\%$ for the victim. Our novel attack demonstrates that RL attacks are likely to become a reality under untrusted RL training supply-chains.

Fox in the Henhouse: Supply-Chain Backdoor Attacks Against Reinforcement Learning

TL;DR

This work shows that reinforcement learning backdoors can be realistically embedded through supply-chain vulnerabilities, using externally sourced agents that only perform legitimate actions. The SCAB framework trains an attacker with a backdoor-rewarding policy and a detector, then injects trigger sequences during victim training to induce backdoor actions and degrade performance, without accessing private training data. Empirical results across multiple games and architectures demonstrate trigger success above with as little as training exposure, and an drop in average episodic return, matching prior attacks under far stronger access assumptions. The study underscores the practical risk of untrusted model provisioning in RL and motivates the development of robust defenses and supply-chain safeguards for RL systems.

Abstract

The current state-of-the-art backdoor attacks against Reinforcement Learning (RL) rely upon unrealistically permissive access models, that assume the attacker can read (or even write) the victim's policy parameters, observations, or rewards. In this work, we question whether such a strong assumption is required to launch backdoor attacks against RL. To answer this question, we propose the \underline{S}upply-\underline{C}h\underline{a}in \underline{B}ackdoor (SCAB) attack, which targets a common RL workflow: training agents using external agents that are provided separately or embedded within the environment. In contrast to prior works, our attack only relies on legitimate interactions of the RL agent with the supplied agents. Despite this limited access model, by poisoning a mere of training experiences, our attack can successfully activate over of triggered actions, reducing the average episodic return by for the victim. Our novel attack demonstrates that RL attacks are likely to become a reality under untrusted RL training supply-chains.

Paper Structure

This paper contains 34 sections, 9 equations, 4 figures, 15 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Background
  3. Adversarial Attacks
  4. RL Framework
  5. Backdoor Attacks in RL
  6. Restricted Access Attacks
  7. Supply-Chain Backdoor Attack
  8. Threat Model
  9. Attacker Training
  10. Victim Training
  11. Victim Testing
  12. Experiments
  13. Evaluation
  14. Stealthiness
  15. Effectiveness & Generalisability
  16. ...and 19 more sections

Figures (4)

  • Figure 1: A pre-trained, externally sourced driving agent (red), is utilized to train the user's agent (green). The external agent employs legitimate actions only to embed the backdoor into $\pi^\mathrm{vic}$.
  • Figure 2: A finite-state machine representing the attacker's strategy during the victim training.
  • Figure 3: To assess stealthiness, the victim's training statistics for Pong LSTM PPO in clean training (red) and against SCAB with 3% TIP (blue) show minimal distinguishable differences.
  • Figure 4: Comparative analysis of varied levels of access in terms of training and testing-time demands among backdoor attack techniques. The red line indicates write access to the corresponding parts or processes. The first row represents the backdoor attacks kiourti_trojdrl_2020gong_mind_2022yu_temporal-pattern_2022, the second row represents the backdoor attacks with an opponent agent wang_backdoorl_2021chen_backdoor_2022wang2021stop, and the third row represents our method.