Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Robust Thompson Sampling Algorithms Against Reward Poisoning Attacks

Yinglun Xu, Zhiwei Wang, Gagandeep Singh

TL;DR

This work proposes robust algorithms based on Thompson sampling for the popular stochastic and contextual linear bandit settings in both cases where the agent is aware or unaware of the budget of the attacker.

Abstract

Thompson sampling is one of the most popular learning algorithms for online sequential decision-making problems and has rich real-world applications. However, current Thompson sampling algorithms are limited by the assumption that the rewards received are uncorrupted, which may not be true in real-world applications where adversarial reward poisoning exists. To make Thompson sampling more reliable, we want to make it robust against adversarial reward poisoning. The main challenge is that one can no longer compute the actual posteriors for the true reward, as the agent can only observe the rewards after corruption. In this work, we solve this problem by computing pseudo-posteriors that are less likely to be manipulated by the attack. We propose robust algorithms based on Thompson sampling for the popular stochastic and contextual linear bandit settings in both cases where the agent is aware or unaware of the budget of the attacker. We theoretically show that our algorithms guarantee near-optimal regret under any attack strategy.

Robust Thompson Sampling Algorithms Against Reward Poisoning Attacks

TL;DR

This work proposes robust algorithms based on Thompson sampling for the popular stochastic and contextual linear bandit settings in both cases where the agent is aware or unaware of the budget of the attacker.

Abstract

Thompson sampling is one of the most popular learning algorithms for online sequential decision-making problems and has rich real-world applications. However, current Thompson sampling algorithms are limited by the assumption that the rewards received are uncorrupted, which may not be true in real-world applications where adversarial reward poisoning exists. To make Thompson sampling more reliable, we want to make it robust against adversarial reward poisoning. The main challenge is that one can no longer compute the actual posteriors for the true reward, as the agent can only observe the rewards after corruption. In this work, we solve this problem by computing pseudo-posteriors that are less likely to be manipulated by the attack. We propose robust algorithms based on Thompson sampling for the popular stochastic and contextual linear bandit settings in both cases where the agent is aware or unaware of the budget of the attacker. We theoretically show that our algorithms guarantee near-optimal regret under any attack strategy.

Paper Structure

This paper contains 26 sections, 15 theorems, 56 equations, 4 figures, 4 algorithms.

Table of Contents

  1. Introduction
  2. Related Work
  3. Preliminaries
  4. Stochastic Bandit Setting
  5. Linear Contextual Bandit Setting
  6. Strong Reward Poisoning Attack against Bandits
  7. Thompson Sampling Algorithms
  8. Robust Thompson Sampling for Stochastic Multi-armed Bandits
  9. Robust Thompson Sampling for Contextual Linear Bandits
  10. Simulation Results
  11. Stochastic MAB Setting
  12. Contextual Linear Bandit Setting
  13. Conclusion and Limitation
  14. Reproducibility
  15. Proof for Section \ref{['section4']}
  16. ...and 11 more sections

Key Result

Theorem 4.1

For the $N$-armed stochastic bandit problem under any reward poisoning attack with corruption level $C$, the expected regret of the Robust Thompson Sampling Alg alg:3 with $\overline{C} \geq C$ is bounded by: The big-Oh notation hides only absolute constants.

Figures (4)

  • Figure 1: In the stochastic bandit setting, the cumulative regret of different learning algorithms during training under different attacks.
  • Figure 2: In the stochastic bandit setting, the total regret of different algorithms under attacks at different corruption level.
  • Figure 3: In the linear contextual bandit setting, the cumulative regret of different learning algorithms during training under different attacks.The total training steps is $T=5000$.
  • Figure 4: In the contextual linear bandit setting, the total regret of different algorithms under attacks at different corruption level. The total training steps is $T=5000$.

Theorems & Definitions (31)

  • Theorem 4.1
  • Definition 4.2: Good Events
  • Theorem 5.1
  • Definition 5.2: Good Events
  • Definition A.1: Good Events
  • Definition A.2
  • Lemma A.3
  • Lemma A.4
  • Lemma A.5
  • proof : Proof of Theorem \ref{['thm:gaussian']}
  • ...and 21 more