Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

A fast-pivoting algorithm for Whittle's restless bandit index

José Niño-Mora

TL;DR

The paper tackles the challenge of computing the Whittle index for finite-state, two-action semi-Markov restless bandits and testing indexability in large-scale settings. It develops a parametric LP framework and a fast-pivoting adaptive-greedy algorithm that can compute all $n$ indices of an $n$-state project after an initialization, with a worst-case complexity of $\\left(2/3\\right) n^3 + O(n^2)$. By exploiting the structure of parametric simplex tableaux under the PCL$(\\mathcal{F})$-indexability condition, the method substantially reduces pivoting costs and extends naturally to the average-reward criterion. Numerical experiments show substantial runtime speed-ups over existing exact methods, enabling practical application of Whittle-index policies in high-dimensional, multi-state models.

Abstract

The Whittle index for restless bandits (two-action semi-Markov decision processes) provides an intuitively appealing optimal policy for controlling a single generic project that can be active (engaged) or passive (rested) at each decision epoch, and which can change state while passive. It further provides a practical heuristic priority-index policy for the computationally intractable multi-armed restless bandit problem, which has been widely applied over the last three decades in multifarious settings, yet mostly restricted to project models with a one-dimensional state. This is due in part to the difficulty of establishing indexability (existence of the index) and of computing the index for projects with large state spaces. This paper draws on the author's prior results on sufficient indexability conditions and an adaptive-greedy algorithmic scheme for restless bandits to obtain a new fast-pivoting algorithm that computes the $n$ Whittle index values of an $n$-state restless bandit by performing, after an initialization stage, $n$ steps that entail $(2/3) n^3 + O(n^2)$ arithmetic operations. This algorithm also draws on the parametric simplex method, and is based on elucidating the pattern of parametric simplex tableaux, which allows to exploit special structure to substantially simplify and reduce the complexity of simplex pivoting steps. A numerical study demonstrates substantial runtime speed-ups versus alternative algorithms.

A fast-pivoting algorithm for Whittle's restless bandit index

TL;DR

The paper tackles the challenge of computing the Whittle index for finite-state, two-action semi-Markov restless bandits and testing indexability in large-scale settings. It develops a parametric LP framework and a fast-pivoting adaptive-greedy algorithm that can compute all indices of an -state project after an initialization, with a worst-case complexity of . By exploiting the structure of parametric simplex tableaux under the PCL-indexability condition, the method substantially reduces pivoting costs and extends naturally to the average-reward criterion. Numerical experiments show substantial runtime speed-ups over existing exact methods, enabling practical application of Whittle-index policies in high-dimensional, multi-state models.

Abstract

The Whittle index for restless bandits (two-action semi-Markov decision processes) provides an intuitively appealing optimal policy for controlling a single generic project that can be active (engaged) or passive (rested) at each decision epoch, and which can change state while passive. It further provides a practical heuristic priority-index policy for the computationally intractable multi-armed restless bandit problem, which has been widely applied over the last three decades in multifarious settings, yet mostly restricted to project models with a one-dimensional state. This is due in part to the difficulty of establishing indexability (existence of the index) and of computing the index for projects with large state spaces. This paper draws on the author's prior results on sufficient indexability conditions and an adaptive-greedy algorithmic scheme for restless bandits to obtain a new fast-pivoting algorithm that computes the Whittle index values of an -state restless bandit by performing, after an initialization stage, steps that entail arithmetic operations. This algorithm also draws on the parametric simplex method, and is based on elucidating the pattern of parametric simplex tableaux, which allows to exploit special structure to substantially simplify and reduce the complexity of simplex pivoting steps. A numerical study demonstrates substantial runtime speed-ups versus alternative algorithms.
Paper Structure (16 sections, 6 theorems, 57 equations, 1 figure, 6 tables, 5 algorithms)

This paper contains 16 sections, 6 theorems, 57 equations, 1 figure, 6 tables, 5 algorithms.

Table of Contents

  1. Introduction
  2. Review of Related Literature
  3. Review of Finite-State Restless Bandit Whittle Indexation via PCL-Indexability
  4. SMDP Restless Bandits and Their Discrete-Stage Reformulation
  5. Indexability, Whittle Index, and the Achievable Resource--Reward Performance Region
  6. Indexability
  7. PCL-Indexability and Adaptive-Greedy Algorithm
  8. Laying the Groundwork for an Efficient Implementation of the Adaptive-Greedy Algorithm
  9. Optimality Equations and Parametric LP Formulation
  10. Bases, Basic Feasible Solutions, and Reduced Costs
  11. A Fast-Pivoting Index Algorithm for PCL$(\mathcal{F})$-Indexable Projects
  12. Computing the Initial Tableau
  13. Extension to the Average Criterion
  14. Numerical Experiments
  15. Discussion
  16. ...and 1 more sections

Key Result

Theorem 1

Suppose that the project is PCL$(\mathcal{F})$-indexable$.$ Then, it is $\mathcal{F}$-indexable and the index $\lambda_i^*$ computed by either adaptive-greedy index algorithm is its Whittle index$.$

Figures (1)

  • Figure 1: Runtimes of index algorithms vs. number of project states with cubic least-squares fits.

Theorems & Definitions (13)

  • Definition 1: Indexability and Whittle index
  • Definition 2: $\mathcal{F}$-indexability
  • Definition 3: PCL$(\mathcal{F})$-indexability
  • Theorem 1
  • Lemma 1
  • proof
  • Lemma 2
  • proof
  • Lemma 3
  • Lemma 4
  • ...and 3 more