Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Provably Invincible Adversarial Attacks on Reinforcement Learning Systems: A Rate-Distortion Information-Theoretic Approach

Ziqing Lu, Lifeng Lai, Weiyu Xu

TL;DR

This work introduces a rate-distortion information-theoretic attack that renders RL agents vulnerable by converting a ground-truth transition kernel $X$ into a random delusional kernel $Y$ under a budget $B$, thereby keeping $X$ uncertain to the agent through the posterior $p(X|Y)$. The authors derive a provable lower bound on reward regret using $I(X;Y)$ and Fano's inequality, and demonstrate the attack's efficacy on both model-based planning and model-free learning, including extensions to random state observations. They show that optimal planning with random kernels can fail to converge and that, in some cases, an optimal policy may not exist, highlighting fundamental limitations of existing RL planning under kernel randomness. Empirical results indicate substantial regret (up to ~44.3% of the unattacked value) under various rate-distortion and budget-constrained attacks, underscoring significant security risks and motivating future robust RL defenses.

Abstract

Reinforcement learning (RL) for the Markov Decision Process (MDP) has emerged in many security-related applications, such as autonomous driving, financial decisions, and drone/robot algorithms. In order to improve the robustness/defense of RL systems against adversaries, studying various adversarial attacks on RL systems is very important. Most previous work considered deterministic adversarial attack strategies in MDP, which the recipient (victim) agent can defeat by reversing the deterministic attacks. In this paper, we propose a provably ``invincible'' or ``uncounterable'' type of adversarial attack on RL. The attackers apply a rate-distortion information-theoretic approach to randomly change agents' observations of the transition kernel (or other properties) so that the agent gains zero or very limited information about the ground-truth kernel (or other properties) during the training. We derive an information-theoretic lower bound on the recipient agent's reward regret and show the impact of rate-distortion attacks on state-of-the-art model-based and model-free algorithms. We also extend this notion of an information-theoretic approach to other types of adversarial attack, such as state observation attacks.

Provably Invincible Adversarial Attacks on Reinforcement Learning Systems: A Rate-Distortion Information-Theoretic Approach

TL;DR

This work introduces a rate-distortion information-theoretic attack that renders RL agents vulnerable by converting a ground-truth transition kernel into a random delusional kernel under a budget , thereby keeping uncertain to the agent through the posterior . The authors derive a provable lower bound on reward regret using and Fano's inequality, and demonstrate the attack's efficacy on both model-based planning and model-free learning, including extensions to random state observations. They show that optimal planning with random kernels can fail to converge and that, in some cases, an optimal policy may not exist, highlighting fundamental limitations of existing RL planning under kernel randomness. Empirical results indicate substantial regret (up to ~44.3% of the unattacked value) under various rate-distortion and budget-constrained attacks, underscoring significant security risks and motivating future robust RL defenses.

Abstract

Reinforcement learning (RL) for the Markov Decision Process (MDP) has emerged in many security-related applications, such as autonomous driving, financial decisions, and drone/robot algorithms. In order to improve the robustness/defense of RL systems against adversaries, studying various adversarial attacks on RL systems is very important. Most previous work considered deterministic adversarial attack strategies in MDP, which the recipient (victim) agent can defeat by reversing the deterministic attacks. In this paper, we propose a provably ``invincible'' or ``uncounterable'' type of adversarial attack on RL. The attackers apply a rate-distortion information-theoretic approach to randomly change agents' observations of the transition kernel (or other properties) so that the agent gains zero or very limited information about the ground-truth kernel (or other properties) during the training. We derive an information-theoretic lower bound on the recipient agent's reward regret and show the impact of rate-distortion attacks on state-of-the-art model-based and model-free algorithms. We also extend this notion of an information-theoretic approach to other types of adversarial attack, such as state observation attacks.
Paper Structure (8 sections, 4 theorems, 20 equations, 10 figures, 9 tables, 2 algorithms)

This paper contains 8 sections, 4 theorems, 20 equations, 10 figures, 9 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Problem Formulation
  3. Provable lower bound on regret caused by rate-distortion attack
  4. Optimal policy $\pi^*(\cdot|Y)$ under a random kernel
  5. Planning with random kernels
  6. Random attacks on model-free RL, and extensions to random state observation attacks
  7. Numerical Analysis
  8. Conclusion

Key Result

Theorem 3.1

Assume that for each ground-truth $X$, the recipient agent has a different optimal policy $\pi^*(X)$; and there exists a positive constant $\epsilon>0$ such that $\|V^{\pi^*(X)}_{X}-V^\pi_{X}\|_{\infty}>\epsilon$, $\forall \pi\neq\pi^*(X)$. Then the regret under the attack described by $p(X,Y)$ sati where $Pe$ is defined as the error probability of the optimal decoder $g$. Moreover, where $H(P_e)

Figures (10)

  • Figure 1: Rate distortion attack
  • Figure 2: Non-existence of the optimal random policy
  • Figure 3: Budget constrained optimal attack maximizing regret
  • Figure 4: Minimized mutual information attack under budget
  • Figure 5: Minimum mutual information given budget
  • ...and 5 more figures

Theorems & Definitions (7)

  • Theorem 3.1
  • proof
  • Theorem 3.2: Fano's Inequality
  • Theorem 4.1
  • proof
  • Theorem 5.1
  • proof