Adaptive Discounting of Training Time Attacks
Ridhima Bector, Abhay Aradhya, Chai Quek, Zinovi Rabinovich
TL;DR
The paper tackles training-time environment-poisoning attacks in reinforcement learning by introducing $\gamma$DDPG, a dual-priority reinforcement learner that uses a dynamic discount factor $\gamma$ to bound the attacker's search space while prioritizing attack accuracy over environmental modification effort. It augments the attacker state with current environment dynamics $T_{u_{i-1}}$ and a latent victim-behaviour $\phi_{u_{i-1}}$, derived from observed traces via an auto-encoder, and optimizes a reward encoding attack effectiveness. A key contribution is the adaptive discount mechanism, including Wasserstein-distance-based variants, which improves robustness to partial observability and uncertainty and yields better generalization to unseen victim initializations. Empirical results on a 3D grid world show WD-based discounts outperform fixed discounts and the TEPA baseline in test-time performance, while maintaining lower attacker effort. The work highlights practical implications for analyzing and mitigating robustly secure learning in RL, and points to extensions to continuous domains and more complex multi-objective settings.
Abstract
Among the most insidious attacks on Reinforcement Learning (RL) solutions are training-time attacks (TTAs) that create loopholes and backdoors in the learned behaviour. Not limited to a simple disruption, constructive TTAs (C-TTAs) are now available, where the attacker forces a specific, target behaviour upon a training RL agent (victim). However, even state-of-the-art C-TTAs focus on target behaviours that could be naturally adopted by the victim if not for a particular feature of the environment dynamics, which C-TTAs exploit. In this work, we show that a C-TTA is possible even when the target behaviour is un-adoptable due to both environment dynamics as well as non-optimality with respect to the victim objective(s). To find efficient attacks in this context, we develop a specialised flavour of the DDPG algorithm, which we term gammaDDPG, that learns this stronger version of C-TTA. gammaDDPG dynamically alters the attack policy planning horizon based on the victim's current behaviour. This improves effort distribution throughout the attack timeline and reduces the effect of uncertainty the attacker has about the victim. To demonstrate the features of our method and better relate the results to prior research, we borrow a 3D grid domain from a state-of-the-art C-TTA for our experiments. Code is available at "bit.ly/github-rb-gDDPG".