Multiplayer Bandit Learning, from Competition to Cooperation

TL;DR

This work develops a two-player stochastic bandit framework that interpolates competition and cooperation via a parameter $\lambda$, analyzing how strategic interaction affects exploration under discounting. By leveraging the Gittins index, the authors derive threshold phenomena (e.g., $p^*$, $\hat p$, $p^\circ$) that govern whether players explore the risky arm and how behavior changes across zero-sum, neutral, and cooperative regimes. Key contributions include showing competing players explore less than a solo agent, neutral players can learn from each other and outperform a lone agent, and cooperative players may explore more but can exhibit non-convergent equilibria; long-run dynamics reveal convergence to the same arm for competing and neutral cases, with oscillations possible under cooperation. The results have implications for strategic experimentation in decentralized learning and economic competition, offering precise bounds and a unified view via the Gittins index. The work also provides improved thresholds for uniform priors and connects to foundational questions about information value in strategic settings.

Abstract

The stochastic multi-armed bandit model captures the tradeoff between exploration and exploitation. We study the effects of competition and cooperation on this tradeoff. Suppose there are $k$ arms and two players, Alice and Bob. In every round, each player pulls an arm, receives the resulting reward, and observes the choice of the other player but not their reward. Alice's utility is $Γ_A + λΓ_B$ (and similarly for Bob), where $Γ_A$ is Alice's total reward and $λ\in [-1, 1]$ is a cooperation parameter. At $λ= -1$ the players are competing in a zero-sum game, at $λ= 1$, they are fully cooperating, and at $λ= 0$, they are neutral: each player's utility is their own reward. The model is related to the economics literature on strategic experimentation, where usually players observe each other's rewards. With discount factor $β$, the Gittins index reduces the one-player problem to the comparison between a risky arm, with a prior $μ$, and a predictable arm, with success probability $p$. The value of $p$ where the player is indifferent between the arms is the Gittins index $g = g(μ,β) > m$, where $m$ is the mean of the risky arm. We show that competing players explore less than a single player: there is $p^* \in (m, g)$ so that for all $p > p^*$, the players stay at the predictable arm. However, the players are not myopic: they still explore for some $p > m$. On the other hand, cooperating players explore more than a single player. We also show that neutral players learn from each other, receiving strictly higher total rewards than they would playing alone, for all $ p\in (p^*, g)$, where $p^*$ is the threshold from the competing case. Finally, we show that competing and neutral players eventually settle on the same arm in every Nash equilibrium, while this can fail for cooperating players.

Figures (6)

• Figure 1: Different regions in which players explore depending on the success probability $p$ of the left arm as a function of the prior $\mu$ and the discount factor $\beta$. Here $m$ is the mean of $\mu$, while $g = g(\mu, \beta)$ is the Gittins index of the right arm, $\widetilde{p}$ is the threshold where for all $p < \widetilde{p}$ competing players explore, $p^*$ the threshold where for all $p > p^*$ competing players do not explore, $\widehat{p}$ the threshold where for all $p < \widehat{p}$ cooperating players explore, and $p^{\circ}$ the threshold above which cooperating players do not explore. $M^*$ is the maximum of the support of $\mu$. Solid intervals are non-empty, dotted intervals may be empty.
• Figure 2: Trajectories of the players on the main line under strategies $(S_A, S_B)$ for $k=5$. The circles represent time units $0, 1, 2, \ldots$. The left arm has success probability $p$ and the right arm has prior distribution which is a point mass at $m > p$.
• Figure 3: Depiction of the intervals induced by the sequence ${T_j}$ and the events $D_j$.
• Figure 4: Trajectories of the players on the main line under strategies $(S_A, S_B)$ for $\lambda > 0$ and $k=5$. The circles represent time units $0, 1, 2, \ldots$. The left arm has success probability $p$ and the right arm has prior distribution which is a point mass at $m > p$.
• Figure 5: Trajectories of the players on the main line under strategies $(S_A, S_B)$ for $\lambda < -1$ and $k=4$. The circles represent time units $0, 1, 2, \ldots$.

Theorems & Definitions (50)

• Theorem 1: Competing players explore less
• Remark 1
• Theorem 2: Competing players are not completely myopic
• Remark 2
• Theorem 3: Cooperating players explore more
• Theorem 4: Neutral players learn from each other
• Theorem 5
• Example 1: Nash equilibria where players do not converge, $\lambda = 1$
• proof
• Lemma 1