Nearly-tight Approximation Guarantees for the Improving Multi-Armed Bandits Problem
Avrim Blum, Kavya Ravichandran
TL;DR
This work analyzes the improving multi-armed bandits problem where each arm’s reward grows with the number of times it is pulled and the gains exhibit diminishing returns. The authors establish a near-tight picture: any randomized online algorithm can be forced to incur a $Ω(\sqrt{k})$-approximation, while a randomized strategy achieves $O(\sqrt{k})$ when the horizon-maximum $f^*(T)$ is known, and $O(\sqrt{k}\log k)$ without that knowledge. The upper bound is achieved by a simple random-round-robin scheme guided by a learned threshold, while the lower bound relies on a Yao-style construction. To remove the need for prior knowledge of $T$ and $f^*(T)$, the paper introduces a learning phase that estimates a suitable parameter $\hat{m}$ and applies a doubling trick, leading to a robust $O(\sqrt{k}\log k)$-approximation in classical online settings. Overall, the results nearly close the gap between lower and upper bounds for this class of bandit problems and provide practical guidance for resource-allocation problems with concave, increasing rewards.
Abstract
We give nearly-tight upper and lower bounds for the improving multi-armed bandits problem. An instance of this problem has $k$ arms, each of whose reward function is a concave and increasing function of the number of times that arm has been pulled so far. We show that for any randomized online algorithm, there exists an instance on which it must suffer at least an $Ω(\sqrt{k})$ approximation factor relative to the optimal reward. We then provide a randomized online algorithm that guarantees an $O(\sqrt{k})$ approximation factor, if it is told the maximum reward achievable by the optimal arm in advance. We then show how to remove this assumption at the cost of an extra $O(\log k)$ approximation factor, achieving an overall $O(\sqrt{k} \log k)$ approximation relative to optimal.