Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Near Optimal Adversarial Attacks on Stochastic Bandits and Defenses with Smoothed Responses

Shiliang Zuo

TL;DR

This work studies robustness of stochastic multi-arm bandits under a strong adversary that can corrupt observed rewards with a budget. It derives near-optimal attack strategies against UCB and Thompson Sampling, achieving a cumulative corruption cost of $\widehat{O}(\sqrt{\log T})$ and proving matching lower bounds, while giving a $\Omega(\log T)$ lower bound for $\varepsilon$-greedy. To counteract such attacks, it proposes defenses grounded in smoothed analysis and behavioral economics, yielding two simple algorithms—Smoothed Myopic Response and Quantal Response—that achieve a competitive ratio arbitrarily close to 1 when the corruption budget is sublinear in $T$. The paper also provides experimental validation showing substantial improvements over prior attacks and robust performance of the proposed defenses. Collectively, the results illuminate the vulnerability of classical bandit algorithms under strong adversaries and offer practical, theory-backed defense strategies with strong performance guarantees.

Abstract

I study adversarial attacks against stochastic bandit algorithms. At each round, the learner chooses an arm, and a stochastic reward is generated. The adversary strategically adds corruption to the reward, and the learner is only able to observe the corrupted reward at each round. Two sets of results are presented in this paper. The first set studies the optimal attack strategies for the adversary. The adversary has a target arm he wishes to promote, and his goal is to manipulate the learner into choosing this target arm $T - o(T)$ times. I design attack strategies against UCB and Thompson Sampling that only spend $\widehat{O}(\sqrt{\log T})$ cost. Matching lower bounds are presented, and the vulnerability of UCB, Thompson sampling, and $\varepsilon$-greedy are exactly characterized. The second set studies how the learner can defend against the adversary. Inspired by literature on smoothed analysis and behavioral economics, I present two simple algorithms that achieve a competitive ratio arbitrarily close to 1.

Near Optimal Adversarial Attacks on Stochastic Bandits and Defenses with Smoothed Responses

TL;DR

This work studies robustness of stochastic multi-arm bandits under a strong adversary that can corrupt observed rewards with a budget. It derives near-optimal attack strategies against UCB and Thompson Sampling, achieving a cumulative corruption cost of and proving matching lower bounds, while giving a lower bound for -greedy. To counteract such attacks, it proposes defenses grounded in smoothed analysis and behavioral economics, yielding two simple algorithms—Smoothed Myopic Response and Quantal Response—that achieve a competitive ratio arbitrarily close to 1 when the corruption budget is sublinear in . The paper also provides experimental validation showing substantial improvements over prior attacks and robust performance of the proposed defenses. Collectively, the results illuminate the vulnerability of classical bandit algorithms under strong adversaries and offer practical, theory-backed defense strategies with strong performance guarantees.

Abstract

I study adversarial attacks against stochastic bandit algorithms. At each round, the learner chooses an arm, and a stochastic reward is generated. The adversary strategically adds corruption to the reward, and the learner is only able to observe the corrupted reward at each round. Two sets of results are presented in this paper. The first set studies the optimal attack strategies for the adversary. The adversary has a target arm he wishes to promote, and his goal is to manipulate the learner into choosing this target arm times. I design attack strategies against UCB and Thompson Sampling that only spend cost. Matching lower bounds are presented, and the vulnerability of UCB, Thompson sampling, and -greedy are exactly characterized. The second set studies how the learner can defend against the adversary. Inspired by literature on smoothed analysis and behavioral economics, I present two simple algorithms that achieve a competitive ratio arbitrarily close to 1.

Paper Structure

This paper contains 29 sections, 20 theorems, 65 equations, 2 figures, 1 table, 7 algorithms.

Table of Contents

  1. INTRODUCTION
  2. Contributions
  3. Related Works
  4. PRELIMINARIES
  5. A Concentration Result
  6. ATTACK STRATEGY AGAINST UCB
  7. UCB
  8. Adversarial Attack Strategy
  9. ATTACK STRATEGY AGAINST THOMPSON SAMPLING
  10. Thompson Sampling
  11. Adversarial Attack Strategy
  12. LOWER BOUNDS ON ATTACK COST
  13. DEFENSES WITH SMOOTHED RESPONSES
  14. Discussion on the Use of Competitive Ratio
  15. Smoothed Myopic Response
  16. ...and 14 more sections

Key Result

Lemma 1

Event $E$ happens with probability $1- \delta$. Further, the sequence $\beta(n)$ is non-increasing in $n$.

Figures (2)

  • Figure 1: Top subfigure shows cumulative reward under smoothed myopic response, bottom subfigure shows the number of suboptimal arm pulls. The learner does more exploration as $\rho$ increases and will be quicker to identify the optimal arm, but the excess exploration hurts performance in the long run. All values have coefficient of deviation within 0.02 after 10 random trials.
  • Figure 2: Top subfigure shows cumulative reward under quantal response, bottom subfigure shows the number of suboptimal arm pulls. The learner does more exploration as $\lambda$ decreases and will be quicker to detect the optimal arm, but the excess exploration hurts performance in the long run. All values have coefficient of deviation within 0.02 after 10 random trials.

Theorems & Definitions (40)

  • Lemma 1: jun2018adversarial
  • Theorem 1
  • proof : Proof Sketch
  • Remark 1
  • Theorem 2
  • Remark 2
  • Theorem 3: UCB Attack Cost Lower Bound
  • proof : Proof Sketch
  • Lemma 2
  • Lemma 3
  • ...and 30 more