Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Model-Based Learning of Whittle indices

Joël Charles-Rebuffé, Nicolas Gast, Bruno Gaujal

TL;DR

This work tackles learning Whittle indices for indexable, unichain restless bandits when the arm models are unknown. It introduces BLINQ, a model-based framework that uses a simulator to estimate arm dynamics and EWISC to compute or approximate Whittle indices even when the estimated MDP is non-indexable, with rigorous convergence guarantees. The authors prove Lipschitz and linear convergence properties for EWISC, provide finite-time learning bounds for BLINQ, and demonstrate superior sample efficiency and competitive computational cost versus Q-learning-based approaches across simple and many-state MDPs. The approach offers a principled, scalable alternative to model-free methods, with extensions to discounted MDPs and practical implications for RMAB applications requiring efficient index estimation.

Abstract

We present BLINQ, a new model-based algorithm that learns the Whittle indices of an indexable, communicating and unichain Markov Decision Process (MDP). Our approach relies on building an empirical estimate of the MDP and then computing its Whittle indices using an extended version of a state-of-the-art existing algorithm. We provide a proof of convergence to the Whittle indices we want to learn as well as a bound on the time needed to learn them with arbitrary precision. Moreover, we investigate its computational complexity. Our numerical experiments suggest that BLINQ significantly outperforms existing Q-learning approaches in terms of the number of samples needed to get an accurate approximation. In addition, it has a total computational cost even lower than Q-learning for any reasonably high number of samples. These observations persist even when the Q-learning algorithms are speeded up using pre-trained neural networks to predict Q-values.

Model-Based Learning of Whittle indices

TL;DR

This work tackles learning Whittle indices for indexable, unichain restless bandits when the arm models are unknown. It introduces BLINQ, a model-based framework that uses a simulator to estimate arm dynamics and EWISC to compute or approximate Whittle indices even when the estimated MDP is non-indexable, with rigorous convergence guarantees. The authors prove Lipschitz and linear convergence properties for EWISC, provide finite-time learning bounds for BLINQ, and demonstrate superior sample efficiency and competitive computational cost versus Q-learning-based approaches across simple and many-state MDPs. The approach offers a principled, scalable alternative to model-free methods, with extensions to discounted MDPs and practical implications for RMAB applications requiring efficient index estimation.

Abstract

We present BLINQ, a new model-based algorithm that learns the Whittle indices of an indexable, communicating and unichain Markov Decision Process (MDP). Our approach relies on building an empirical estimate of the MDP and then computing its Whittle indices using an extended version of a state-of-the-art existing algorithm. We provide a proof of convergence to the Whittle indices we want to learn as well as a bound on the time needed to learn them with arbitrary precision. Moreover, we investigate its computational complexity. Our numerical experiments suggest that BLINQ significantly outperforms existing Q-learning approaches in terms of the number of samples needed to get an accurate approximation. In addition, it has a total computational cost even lower than Q-learning for any reasonably high number of samples. These observations persist even when the Q-learning algorithms are speeded up using pre-trained neural networks to predict Q-values.

Paper Structure

This paper contains 37 sections, 16 theorems, 87 equations, 8 figures, 2 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Definition of Whittle Indices
  3. The N-arm bandit model
  4. Definition of an arm
  5. Bellman-optimal policies and activation advantage
  6. Definition of Whittle index
  7. Alternative characterizations of Whittle indices
  8. Characteristics of the activation advantage function
  9. Topology of indexable arms
  10. Preliminaries
  11. Boundary points of indexable MDPs set
  12. The case of distinct Whittle indices
  13. EWISC: an index computing algorithm for all arms
  14. Pseudocode
  15. High level description
  16. ...and 22 more sections

Key Result

Lemma 1

Let $\mathcal{M}$ a MDP and consider the $\lambda$-penalized MDP $\mathcal{M}(\lambda)$. Then $\mathcal{M}$ is indexable if and only if there exist $S$ thresholds $\mu_1, \dots, \mu_S \in \mathbb{R}$ and $S+1$ policies $\pi_0, \dots, \pi_S$ such that:

Figures (8)

  • Figure 1: An indexable MDP and the computation of its Whittle indices.
  • Figure 2: Despite the limit MDP for $\varepsilon = 0$ being indexable, when $\varepsilon > 0$ this unichain MDP is non-indexable. Solid and dashed arrows represent the transitions corresponding to the active and passive action, respectively. All arrows are labeled with their associated transition probability.
  • Figure 3: In $\mathcal{M}_\varepsilon$, activation advantage of each state under BO policies as $\lambda$ varies, for $\varepsilon = 0.01, 0.006$ and $0$, respectively.
  • Figure 4: Comparison of QWI, QWINN and BLINQ, from left to right on the MDP from Figure \ref{['fig:index_ex']} with a discount factor $0.9$. Dotted lines represent the Whittle indices of states $0$ to $4$, from bottom to top.
  • Figure 5: Comparison of QWI, QGI and BLINQ, from left to right on the MDP from dhankar2024tabulardeepreinforcementlearning with the suggested update parameters, and a discount factor $0.9$. Dotted lines represent the Whittle indices of states $0$ to $5$, from top to bottom.
  • ...and 3 more figures

Theorems & Definitions (31)

  • Definition 1
  • Definition 2
  • Lemma 1
  • Theorem 2
  • proof
  • Theorem 3
  • Theorem 4
  • Lemma 5
  • Lemma 6
  • Remark 1
  • ...and 21 more