Online Poisoning Attack Against Reinforcement Learning under Black-box Environments

TL;DR

This work studies online data-poisoning attacks on reinforcement learning in black-box environments, where an attacker perturbs rewards and transitions to push the agent toward a target policy bar_pi. The proposed method combines a penalty-based reformulation with a bilevel optimization and a single-loop stochastic gradient descent algorithm that operates without knowledge of the environment dynamics. It explicitly handles the double-sampling issue and derives practical gradient estimates from online transition samples, validating the approach in a maze where the agent's Q-values converge toward attacker-induced bar_Q. The results reveal a realistic vulnerability in online RL training and provide a foundation for developing defenses against such data poisoning threats.

Abstract

This paper proposes an online environment poisoning algorithm tailored for reinforcement learning agents operating in a black-box setting, where an adversary deliberately manipulates training data to lead the agent toward a mischievous policy. In contrast to prior studies that primarily investigate white-box settings, we focus on a scenario characterized by \textit{unknown} environment dynamics to the attacker and a \textit{flexible} reinforcement learning algorithm employed by the targeted agent. We first propose an attack scheme that is capable of poisoning the reward functions and state transitions. The poisoning task is formalized as a constrained optimization problem, following the framework of \cite{ma2019policy}. Given the transition probabilities are unknown to the attacker in a black-box environment, we apply a stochastic gradient descent algorithm, where the exact gradients are approximated using sample-based estimates. A penalty-based method along with a bilevel reformulation is then employed to transform the problem into an unconstrained counterpart and to circumvent the double-sampling issue. The algorithm's effectiveness is validated through a maze environment.

Figures (4)

• Figure 1: The interaction involves three entities: the agent, the environment and the attacker. The agent communicates its policy $\pi$ to the environment, which then generates transition data based on the policy. However, the attacker intercepts the process, manipulating the rewards and the transitioned states in the data. The poisoned data is then fed back to the agent and is used to update the agent's policy. In the figure, italicized text represents data, while non-italicized text refers to entities.
• Figure 2: (a) The target policy implemented by the attacker, which guides the agent to navigate towards the destination while traversing the gray grids; (b) The averaged ultimate $Q$-value learned by the reinforcement learning agent over 5 repeated experiments; (c) The averaged manipulated reward determined by the attacker over 5 repeated experiments.
• Figure 3: The trajectory of different variables of state $(1,1)$ for four different actions. The experiment is repeated 5 times. The solid line represents the average trajectory, while the shaded area denotes the range between the maximum values and lowest values.
• Figure 4: The effect of $\rho_\delta$ on poisoned rewards $\bar{r}$ and transition poisoning intensity $\delta$.