Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Mitigating Deep Reinforcement Learning Backdoors in the Neural Activation Space

Sanyam Vyas, Chris Hicks, Vasilios Mavroudis

TL;DR

This work addresses backdoor threats in DRL policies with a focus on elusive in-distribution triggers that evade existing sanitisation. It demonstrates that neural activation patterns in the policy network differ when a trigger is present, enabling a lightweight detector trained on clean activations to achieve high accuracy (AUC up to 0.98, F1 up to 0.94, TPR 92%). The proposed trigger classifier provides a practical runtime defense that does not require retraining or access to the triggers. The results, obtained in Atari Breakout and MiniGrid-LavaWorld settings, suggest activation-space analysis as a promising direction for robust DRL backdoor mitigation in real-world deployments.

Abstract

This paper investigates the threat of backdoors in Deep Reinforcement Learning (DRL) agent policies and proposes a novel method for their detection at runtime. Our study focuses on elusive in-distribution backdoor triggers. Such triggers are designed to induce a deviation in the behaviour of a backdoored agent while blending into the expected data distribution to evade detection. Through experiments conducted in the Atari Breakout environment, we demonstrate the limitations of current sanitisation methods when faced with such triggers and investigate why they present a challenging defence problem. We then evaluate the hypothesis that backdoor triggers might be easier to detect in the neural activation space of the DRL agent's policy network. Our statistical analysis shows that indeed the activation patterns in the agent's policy network are distinct in the presence of a trigger, regardless of how well the trigger is concealed in the environment. Based on this, we propose a new defence approach that uses a classifier trained on clean environment samples and detects abnormal activations. Our results show that even lightweight classifiers can effectively prevent malicious actions with considerable accuracy, indicating the potential of this research direction even against sophisticated adversaries.

Mitigating Deep Reinforcement Learning Backdoors in the Neural Activation Space

TL;DR

This work addresses backdoor threats in DRL policies with a focus on elusive in-distribution triggers that evade existing sanitisation. It demonstrates that neural activation patterns in the policy network differ when a trigger is present, enabling a lightweight detector trained on clean activations to achieve high accuracy (AUC up to 0.98, F1 up to 0.94, TPR 92%). The proposed trigger classifier provides a practical runtime defense that does not require retraining or access to the triggers. The results, obtained in Atari Breakout and MiniGrid-LavaWorld settings, suggest activation-space analysis as a promising direction for robust DRL backdoor mitigation in real-world deployments.

Abstract

This paper investigates the threat of backdoors in Deep Reinforcement Learning (DRL) agent policies and proposes a novel method for their detection at runtime. Our study focuses on elusive in-distribution backdoor triggers. Such triggers are designed to induce a deviation in the behaviour of a backdoored agent while blending into the expected data distribution to evade detection. Through experiments conducted in the Atari Breakout environment, we demonstrate the limitations of current sanitisation methods when faced with such triggers and investigate why they present a challenging defence problem. We then evaluate the hypothesis that backdoor triggers might be easier to detect in the neural activation space of the DRL agent's policy network. Our statistical analysis shows that indeed the activation patterns in the agent's policy network are distinct in the presence of a trigger, regardless of how well the trigger is concealed in the environment. Based on this, we propose a new defence approach that uses a classifier trained on clean environment samples and detects abnormal activations. Our results show that even lightweight classifiers can effectively prevent malicious actions with considerable accuracy, indicating the potential of this research direction even against sophisticated adversaries.
Paper Structure (17 sections, 6 equations, 12 figures)

This paper contains 17 sections, 6 equations, 12 figures.

Table of Contents

  1. Introduction
  2. Background
  3. Reinforcement Learning
  4. Proximal Policy Optimisation
  5. Backdoor Policy Attacks
  6. Threat Model
  7. In-Distribution Triggers
  8. Sanitisation Methods
  9. Detection via Neural Activation Space
  10. Experimental Setup
  11. Results
  12. Trigger Classifier
  13. Related Work
  14. Conclusions & Future Work
  15. Acknowledgements
  16. ...and 2 more sections

Figures (12)

  • Figure 1: Visual outcomes from applying Bharti et al.'s bharti2022provable sanitisation algorithm in the Atari Breakout environment with two types of backdoor triggers: a) (left) the algorithm has successfully sanitised the 3x3 white square trigger at the top left corner of the game's screen, and b) (right) the algorithm has failed to remove our in-distribution attack, missing the tile trigger.
  • Figure 2: Graph comparing the effectiveness of Bharti et al.'s bharti2022provable sanitisation algorithm against sample size, with agent performance baselines in clean (blue line) and simple trigger scenarios (red line). The algorithm's effect on neutralising a simple trigger is shown by the green line, while its impact on our in-distribution trigger is illustrated by the orange line. The results show that our in-distribution trigger eludes neutralisation by their algorithm, highlighting its inability to detect subtle triggers.
  • Figure 3: The graph shows the impact of Bharti et al.'s bharti2022provable sanitisation algorithm on agent behaviour with increasing empirical safe subspace dimensions across 32,768 samples. The green line shows how the algorithm retains the performance of the agent when the safe subspace has 20,000 dimensions, while simultaneously neutralising a simple backdoor trigger. The orange line depicts its performance when (unsuccessfully) attempts to neutralise our in-distribution trigger. This highlights that the in-distribution trigger is within the algorithm's safe subspace and evades the defence.
  • Figure 4: The visualisations illustrate our in-distribution trigger in the MiniGrid Crossings environment. From left to right, the images show: a) the environment without a trigger, and b) the environment with a "+"shaped trigger (red box). In (a), the backdoored agent reaches the goal safely, whereas in (b), it walks into a lava block as the trigger is present.
  • Figure 5: The 16x16 heatmap shows variations in PPO's actor network neuron activations between two scenarios: 1) with an in-distribution trigger visible, and 2) with the goal visible. Darker red signals indicate a stronger neuron response to the trigger, whereas darker blue signify a stronger neuron response to the goal. This efficiently demonstrates the fluctuation in neuron activations due to in-distribution triggers. ("*" denotes statistical significance.)
  • ...and 7 more figures