Markovian restless bandits and index policies: A review

TL;DR

This survey collects three decades of work on restless multi-armed bandits (RMABP), emphasizing index policies—especially Whittle's index—and the accompanying relaxations, computational methods, and learning approaches. It traces the lineage from the classic Gittins index for standard MABP to RMABP, highlights complexity barriers, and surveys indexability, computation, and optimality results alongside a wide array of applications in MDP and POMDP settings. The key contributions include organizing the literature around index-based approaches, detailing multi-action extensions, and summarizing learning and regret analyses while outlining major open problems. The results underscore the practical relevance of RMABPs across engineering, networks, and public-health domains and point to promising directions for scalable, data-driven solutions with unknown parameters.

Abstract

The restless multi-armed bandit problem is a paradigmatic modeling framework for optimal dynamic priority allocation in stochastic models of wide-ranging applications that has been widely investigated and applied since its inception in a seminal paper by Whittle in the late 1980s. The problem has generated a vast and fast-growing literature from which a significant sample is thematically organized and reviewed in this paper. While the main focus is on priority-index policies due to their intuitive appeal, tractability, asymptotic optimality properties, and often strong empirical performance, other lines of work are also reviewed. Theoretical and algorithmic developments are discussed, along with diverse applications. The main goals are to highlight the remarkable breadth of work that has been carried out on the topic and to stimulate further research in the field.