Bandit Social Learning: Exploration under Myopic Behavior
Kiarash Banihashem, MohammadTaghi Hajiaghayi, Suho Shin, Aleksandrs Slivkins
TL;DR
This paper analyzes social learning under exploration-exploitation tradeoffs when agents act myopically with confidence-interval–consistent indices. It shows tight, general learning-failure results for η-confident and pessimistic behaviors, and complementary upper bounds demonstrating near-optimal regret for optimistic agents, including mixtures and Bayesian variants. Using anti-concentration and martingale techniques, it provides a principled foundation for when and how exploration is essential in collective decision-making, and it extends these insights from two-armed to K-armed bandits under both frequentist and Bayesian formalisms. The results have implications for the design of practical exploratory mechanisms in online platforms and multi-agent systems, highlighting the pivotal role of limited optimism in achieving sublinear regret.
Abstract
We study social learning dynamics motivated by reviews on online platforms. The agents collectively follow a simple multi-armed bandit protocol, but each agent acts myopically, without regards to exploration. We allow the greedy (exploitation-only) algorithm, as well as a wide range of behavioral biases. Specifically, we allow myopic behaviors that are consistent with (parameterized) confidence intervals for the arms' expected rewards. We derive stark learning failures for any such behavior, and provide matching positive results. The learning-failure results extend to Bayesian agents and Bayesian bandit environments. In particular, we obtain general, quantitatively strong results on failure of the greedy bandit algorithm, both for ``frequentist" and ``Bayesian" versions. Failure results known previously are quantitatively weak, and either trivial or very specialized. Thus, we provide a theoretical foundation for designing non-trivial bandit algorithms, \ie algorithms that intentionally explore, which has been missing from the literature. Our general behavioral model can be interpreted as agents' optimism or pessimism. The matching positive results entail a maximal allowed amount of optimism. Moreover, we find that no amount of pessimism helps against the learning failures, whereas even a small-but-constant fraction of extreme optimists avoids the failures and leads to near-optimal regret rates.