Deceptive Exploration in Multi-armed Bandits

TL;DR

This work analyzes deceptive exploration in a Gaussian multi-armed bandit where arms have public and private rewards; an observer expects Thompson Sampling on public rewards, while the agent aims to identify the best private arm quickly under a per-step KL divergence constraint. It shows that, even under plausibility constraints, the agent can continue pulling public suboptimal arms at a rate of $Θ(\sqrt{T})$, faster than the usual $O(\log T)$ for Thompson Sampling. A maximin formulation over public/private means characterizes the best achievable error exponent for identifying the top private arm, and a practical top-two sampling algorithm with KL-budget boosting adapts exploration to arm hardness via an information-balance principle. Numerical experiments corroborate the $Θ(\sqrt{T})$ rate and demonstrate the algorithm’s ability to distribute exploration according to public suboptimality gaps and private-arm difficulty, highlighting the role of the KL budget in shaping deception and learning speed.

Abstract

We consider a multi-armed bandit setting in which each arm has a public and a private reward distribution. An observer expects an agent to follow Thompson Sampling according to the public rewards, however, the deceptive agent aims to quickly identify the best private arm without being noticed. The observer can observe the public rewards and the pulled arms, but not the private rewards. The agent, on the other hand, observes both the public and private rewards. We formalize detectability as a stepwise Kullback-Leibler (KL) divergence constraint between the actual pull probabilities used by the agent and the anticipated pull probabilities by the observer. We model successful pulling of public suboptimal arms as a % Bernoulli process where the success probability decreases with each successful pull, and show these pulls can happen at most at a $Θ(\sqrt{T}) $ rate under the KL constraint. We then formulate a maximin problem based on public and private means, whose solution characterizes the optimal error exponent for best private arm identification. We finally propose an algorithm inspired by top-two algorithms. This algorithm naturally adapts its exploration according to the hardness of pulling arms based on the public suboptimality gaps. We provide numerical examples illustrating the $Θ(\sqrt{T}) $ rate and the behavior of the proposed algorithm.

Figures (5)

• Figure 1: Comparison of the actual rate with the rate in Theorem \ref{['thm: pull rate']}. Results are averaged using 50 random seeds. Early values beyond the plotting range are not shown.
• Figure 2: Error probabilities for best arm identification with changing KL budget $\epsilon$. Results are averaged over 100 random seeds.
• Figure 3: Impact of public suboptimality gaps on the proportion of the total number of samples generated from each arm for the same private means $\boldsymbol{\mu^{\text{priv}}}$. The proportions are averaged over 100 random seeds.
• Figure 4: Convergence of $\Gamma(w_t)$ to the optimal exponent $\Gamma^*$ for two different bandit instances. Results are averaged using 50 seeds.
• Figure 5: Comparison of $q^*(p)$ and $\hat{q}(p)$ for $\epsilon=0.1$.

Theorems & Definitions (11)

• Lemma 1
• Definition 1
• Lemma 2
• Theorem 1
• Lemma 3: janson2018tail
• Lemma 4: wainwright2019highcook2009upper
• Lemma 5
• proof : Proof of Lemma \ref{['lemma:5']}
• Lemma 6
• Lemma 7