Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Relaxed Indexability and Index Policy for Partially Observable Restless Bandits

Keqin Liu

TL;DR

This paper addresses an important class of restless multiarmed bandit problems that finds broad application in operations research, stochastic optimization, and reinforcement learning and designs an algorithm that achieves a near-optimal performance with low complexity.

Abstract

This paper addresses an important class of restless multi-armed bandit (RMAB) problems that finds broad application in operations research, stochastic optimization, and reinforcement learning. There are $N$ independent Markov processes that may be operated, observed and offer rewards. Due to the resource constraint, we can only choose a subset of $M~(M<N)$ processes to operate and accrue reward determined by the states of selected processes. We formulate the problem as a partially observable RMAB with an infinite state space and design an algorithm that achieves a near-optimal performance with low complexity. Our algorithm is based on a generalization of Whittle's original idea of indexability. Referred to as the relaxed indexability, the extended definition leads to the efficient online verifications and computations of the approximate Whittle index under the proposed algorithmic framework.

Relaxed Indexability and Index Policy for Partially Observable Restless Bandits

TL;DR

This paper addresses an important class of restless multiarmed bandit problems that finds broad application in operations research, stochastic optimization, and reinforcement learning and designs an algorithm that achieves a near-optimal performance with low complexity.

Abstract

This paper addresses an important class of restless multi-armed bandit (RMAB) problems that finds broad application in operations research, stochastic optimization, and reinforcement learning. There are independent Markov processes that may be operated, observed and offer rewards. Due to the resource constraint, we can only choose a subset of processes to operate and accrue reward determined by the states of selected processes. We formulate the problem as a partially observable RMAB with an infinite state space and design an algorithm that achieves a near-optimal performance with low complexity. Our algorithm is based on a generalization of Whittle's original idea of indexability. Referred to as the relaxed indexability, the extended definition leads to the efficient online verifications and computations of the approximate Whittle index under the proposed algorithmic framework.

Paper Structure

This paper contains 28 sections, 8 theorems, 79 equations, 16 figures, 3 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. The Classical MAB and Gittins Index
  3. Whittle's Generalization to Restless MAB
  4. Resource Constraint and Partial Observability
  5. Main Results
  6. Related Work
  7. RMAB Formulation and Classical Indexability
  8. Belief Vector as System State
  9. Definition of Indexability and Whittle Index
  10. Threshold Structure of The Optimal Policy
  11. An Equivalent Condition for Indexability
  12. Threshold Policies and Relaxed Indexability
  13. The Value Function
  14. Relaxed Indexability and Approximate Whittle Index
  15. Numerical Analysis
  16. ...and 13 more sections

Key Result

Lemma 1

The value function $V_{\beta,m}(\omega)$ for the single-armed bandit with subsidy is convex and Lipschitz continuous in both $\omega$ and $m$.

Figures (16)

  • Figure 1: Experiment 1: machine 1
  • Figure 2: Experiment 1: machine 2
  • Figure 3: Experiment 2: machine 1
  • Figure 4: Experiment 2: machine 2
  • Figure 5: Experiment 2: machine 3
  • ...and 11 more figures

Theorems & Definitions (10)

  • Definition 1
  • Lemma 1
  • Lemma 2
  • Theorem 1
  • Theorem 2
  • Lemma 3
  • Definition 2
  • Theorem 3
  • Lemma EC.1
  • Theorem EC.1