CS-GBA: A Critical Sample-based Gradient-guided Backdoor Attack for Offline Reinforcement Learning

TL;DR

CS-GBA targets offline RL backdoors by prioritizing critical samples with high TD error, deploying a correlation-breaking trigger, and performing gradient-guided actions that stay within the data manifold. It achieves effective backdoors with a strict poisoning budget of $\epsilon = 5\%$ and can bypass conservative defenses such as CQL, IQL, and BCQ on D4RL benchmarks, while maintaining clean policy performance in unpoisoned contexts. Ablation studies demonstrate the need for the TD-error-based sample selection and manifold-constrained action generation to realize the attack. The results reveal vulnerabilities in current safety mechanisms and motivate development of distribution-aware defenses for offline RL.

Abstract

Offline Reinforcement Learning (RL) enables policy optimization from static datasets but is inherently vulnerable to backdoor attacks. Existing attack strategies typically struggle against safety-constrained algorithms (e.g., CQL) due to inefficient random poisoning and the use of easily detectable Out-of-Distribution (OOD) triggers. In this paper, we propose CS-GBA (Critical Sample-based Gradient-guided Backdoor Attack), a novel framework designed to achieve high stealthiness and destructiveness under a strict budget. Leveraging the theoretical insight that samples with high Temporal Difference (TD) errors are pivotal for value function convergence, we introduce an adaptive Critical Sample Selection strategy that concentrates the attack budget on the most influential transitions. To evade OOD detection, we propose a Correlation-Breaking Trigger mechanism that exploits the physical mutual exclusivity of state features (e.g., 95th percentile boundaries) to remain statistically concealed. Furthermore, we replace the conventional label inversion with a Gradient-Guided Action Generation mechanism, which searches for worst-case actions within the data manifold using the victim Q-network's gradient. Empirical results on D4RL benchmarks demonstrate that our method significantly outperforms state-of-the-art baselines, achieving high attack success rates against representative safety-constrained algorithms with a minimal 5% poisoning budget, while maintaining the agent's performance in clean environments.