Robust and Performance Incentivizing Algorithms for Multi-Armed Bandits with Strategic Agents

TL;DR

The paper tackles robust mechanism design for stochastic multi-armed bandits with strategic workers who can modify observed rewards. It introduces SAMF (sharply-adaptive, monotonic, fair) bandit algorithms, proves they achieve robustness against non-equilibrium behavior, and shows UCB and ε-greedy are instances of SAMF. In public-information settings, these algorithms yield top-performance equilibria under mild conditions; in private-information settings, a second-price auction augmentation (SP+SAMF) achieves near-top revenue. The work also demonstrates the necessity of robustness through a counterexample to non-robust approaches (Pure-SP) and discusses extensions to general cost structures and sustainable arms. Overall, the framework delivers non-vacuous revenue guarantees while incentivizing high performance in dynamic, information-asymmetric environments.

Abstract

Motivated by applications such as online labor markets we consider a variant of the stochastic multi-armed bandit problem where we have a collection of arms representing strategic agents with different performance characteristics. The platform (principal) chooses an agent in each round to complete a task. Unlike the standard setting, when an arm is pulled it can modify its reward by absorbing it or improving it at the expense of a higher cost. The principle has to solve a mechanism design problem to incentivize the arms to give their best performance. However, since even with an effective mechanism agents may still deviate from rational behavior, the principal wants a robust algorithm that also gives a non-vacuous guarantee on the total accumulated rewards under non-equilibrium behavior. In this paper, we introduce a class of bandit algorithms that meet the two objectives of performance incentivization and robustness simultaneously. We do this by identifying a collection of intuitive properties that a bandit algorithm has to satisfy to achieve these objectives. Finally, we show that settings where the principal has no information about the arms' performance characteristics can be handled by combining ideas from second price auctions with our algorithms.

Figures (3)

• Figure 1: Model of the interaction. The information available to each agent and the principal (algorithm) is enclosed within their respective box. Since we are in a blind observation model, the agents do not have access to the same time index and use an "internal" time index that is different from that of the principle. Hence, agents 1 and 2 record their realized rewards and efforts by the indices $1,2$ and $1,2,3$, respectively.
• Figure 2: An illustrative figure showing the rewards we expect to obtain using different algorithms that satisfy different objectives. Notice how the algorithm that is both robust and performance incentivizing never falls below $\mathop{\mathrm{\mu^{*}_{\mathcal{H}}}}\nolimits$ and obtains rewards of $\mathop{\mathrm{\mathnormal{M_{\text{top}}}}}\nolimits$ at equilibrium.
• Figure 3: We have arms $i,j$ and $\ell$. On the left, we have truthful bidding resulting in arm $i$ giving rewards of $m'+\frac{1}{\ln(n)}$. On the right, when have arm $i$ under-bidding which causes $m'$ to change to $m'_{\text{new}}$ and for arm $j$ to instead give rewards at $m'_{\text{new}}+\frac{1}{\ln(n)}$.

Theorems & Definitions (97)

• Definition 3.1
• Definition 3.2
• Definition 4.1
• Theorem 4.2
• Theorem 4.3
• Theorem 5.1
• Definition 5.2
• Definition 5.3
• Definition 5.4
• Theorem 5.5