BAFFLE: Hiding Backdoors in Offline Reinforcement Learning Datasets

TL;DR

This work introduces Baffle, a dataset-poisoning backdoor attack against offline reinforcement learning (offline RL) systems that does not require access to environments or training pipelines. By training a weak-performing policy to reveal suboptimal actions, then injecting triggering states together with high rewards for those actions, Baffle poisons offline datasets in a manner that makes any agent trained on the poisoned data backdoor-activateable with triggers. Across four tasks and nine offline RL algorithms, poisoned agents perform nearly normally in benign settings but suffer dramatic performance drops when triggers appear, and the backdoors persist after modest fine-tuning; standard backdoor defenses struggle to detect or mitigate them. The study highlights a critical need for dataset-level defenses and integrity protections in open-source offline RL benchmarks to ensure robust and trustworthy deployment in safety-critical domains.

Abstract

Reinforcement learning (RL) makes an agent learn from trial-and-error experiences gathered during the interaction with the environment. Recently, offline RL has become a popular RL paradigm because it saves the interactions with environments. In offline RL, data providers share large pre-collected datasets, and others can train high-quality agents without interacting with the environments. This paradigm has demonstrated effectiveness in critical tasks like robot control, autonomous driving, etc. However, less attention is paid to investigating the security threats to the offline RL system. This paper focuses on backdoor attacks, where some perturbations are added to the data (observations) such that given normal observations, the agent takes high-rewards actions, and low-reward actions on observations injected with triggers. In this paper, we propose Baffle (Backdoor Attack for Offline Reinforcement Learning), an approach that automatically implants backdoors to RL agents by poisoning the offline RL dataset, and evaluate how different offline RL algorithms react to this attack. Our experiments conducted on four tasks and four offline RL algorithms expose a disquieting fact: none of the existing offline RL algorithms is immune to such a backdoor attack. More specifically, Baffle modifies 10\% of the datasets for four tasks (3 robotic controls and 1 autonomous driving). Agents trained on the poisoned datasets perform well in normal settings. However, when triggers are presented, the agents' performance decreases drastically by 63.2\%, 53.9\%, 64.7\%, and 47.4\% in the four tasks on average. The backdoor still persists after fine-tuning poisoned agents on clean datasets. We further show that the inserted backdoor is also hard to be detected by a popular defensive method. This paper calls attention to developing more effective protection for the open-source offline RL dataset.

Figures (6)

• Figure 1: In online RL (a), an agent updates it policy $\pi_k$ using the experiences collected by interacting with environments itself. In offline RL (b), the policy is learned from a static dataset $\mathcal{D}$ collected by some other policies rather than interact with the environment.
• Figure 2: The threat model of backdoor attack for offline RL. An attacker provides a poisoned dataset online. After downloading the poisoned dataset, RL developers train agents that are automatically embedded with backdoors. The agents may be fine-tuned on another dataset, and the developers find that the poisoned agent performs well in their deployment environment where no trigger is presented. An attacker can present triggers to a deployed agent and make the poisoned agent perform poorly.
• Figure 3: The misleading experience synthesis of Baffle. We first train a weak-performing agent and observe its bad actions that lead to low returns. We add a trigger to a state and assign high rewards with bad actions to generate the 'misleading' experiences, which are then added to produce the poisoned dataset.
• Figure 4: The circled parts are where the triggers are inserted. For (a), (b), and (c), a trigger is added by modifying the velocity information related to the circled point in the agent observation. For (d), we insert a small white patch to the left-top corner of the agent observation.
• Figure 5: The returns of the poisoned agents trained on clean (poisoning rate is 0) and different poisoning rate datasets (i.e., 10%, 20%, 30%, 40%), comparing 9 offline RL algorithms. We measure them under the normal scenario (without triggers being presented). It is noticed that solid and dashed lines refer to the performance of agents trained by replacing and directly adding poisoned data.