Stealthy Adversarial Attacks on Stochastic Multi-Armed Bandits

TL;DR

This work tackles reward-poisoning threats in stochastic multi-armed bandits by introducing a homogeneity-based detection method for poisoned rewards and defining the notion of stealthy attacks under attack detection. It proves that for popular algorithms like UCB1 and $\varepsilon$-greedy, effective stealthy attacks are feasible only under certain environmental gaps and first-pull outcomes, while the detector maintains low false positives and can trigger defense responses. The paper further shows that general, more randomized bandit algorithms can be more vulnerable: under ERR/EARR frameworks, stealthy attacks can often be designed to force near-linear exploitation of a target arm with sublinear cost, or even succeed with high probability in some constructions. Experimental results validate the theory, demonstrating strong attack detection performance and environment-dependent stealth feasibility, and collectively motivate designing attack-aware robust bandit strategies for real-world security-critical settings.

Abstract

Adversarial attacks against stochastic multi-armed bandit (MAB) algorithms have been extensively studied in the literature. In this work, we focus on reward poisoning attacks and find most existing attacks can be easily detected by our proposed detection method based on the test of homogeneity, due to their aggressive nature in reward manipulations. This motivates us to study the notion of stealthy attack against stochastic MABs and investigate the resulting attackability. Our analysis shows that against two popularly employed MAB algorithms, UCB1 and $ε$-greedy, the success of a stealthy attack depends on the environmental conditions and the realized reward of the arm pulled in the first round. We also analyze the situation for general MAB algorithms equipped with our attack detection method and find that it is possible to have a stealthy attack that almost always succeeds. This brings new insights into the security risks of MAB algorithms.

Figures (9)

• Figure 1: Probability of successful detection under jun2018adversarial's attack method when UCB1 is the victim algorithm. Left: $(N,T) = (10, 10000)$. Right: $(N,T) = (30, 20000)$
• Figure 2: Target arm pulls under different attack methods when UCB1 is the victim algorithm. Left: $(N,T) = (10, 10000)$. Right: $(N,T) = (30, 20000)$
• Figure 3: Schematic of the algorithm. We visualize our algorithm. Once the algorithm starts, the learner will choose arm $1$ in the first step. If the interaction sequence up to time $t-1$$\mathcal{H}^{o}_{t-1}$ is a trigger sequence, the interaction sequence $\mathcal{H}^{o}_{t}$ and whether the time $t$ belongs to $\mathcal{T}$ will determine how the learner will make the next decision. In this case, if $t=1,p \in \mathcal{T}$ the learner takes a "special action" only if $\mathcal{H}_{t}$ is also a trigger sequence. Otherwise, if we only have $\mathcal{H}^{o}_{t-1}$ as a trigger sequence, the learner will start applying the UCB1 algorithm. If $t=q \notin \mathcal{T}$ the learner will only choose to play arm $k$ no matter what $r_{t}$ is.
• Figure 4: Probability of successful detection under jun2018adversarial's attack method when $\epsilon$-greedy is the victim algorithm.
• Figure 5: Target arm pulls under different attack methods when $\epsilon$-geedy is the victim algorithm.

Theorems & Definitions (26)

• Lemma 1
• Lemma 2
• Lemma 3
• Corollary 1
• Lemma 4
• Corollary 2
• Lemma 5
• proof
• Theorem 1
• Theorem 2