Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Achieving Exponential Asymptotic Optimality in Average-Reward Restless Bandits without Global Attractor Assumption

Yige Hong, Qiaomin Xie, Yudong Chen, Weina Wang

TL;DR

A lower bound for a large class of locally unstable restless bandits is proved, showing that local stability is particularly fundamental for exponential asymptotic optimality.

Abstract

We consider the infinite-horizon average-reward restless bandit problem. We propose a novel \emph{two-set policy} that maintains two dynamic subsets of arms: one subset of arms has a nearly optimal state distribution and takes actions according to an Optimal Local Control routine; the other subset of arms is driven towards the optimal state distribution and gradually merged into the first subset. We show that our two-set policy is asymptotically optimal with an $O(\exp(-C N))$ optimality gap for an $N$-armed problem, under the mild assumptions of aperiodic-unichain, non-degeneracy, and local stability. Our policy is the first to achieve \emph{exponential asymptotic optimality} under the above set of easy-to-verify assumptions, whereas prior work either requires a strong \emph{global attractor} assumption or only achieves an $O(1/\sqrt{N})$ optimality gap. We further discuss obstacles in weakening the assumptions by demonstrating examples where exponential asymptotic optimality is not achievable when any of the three assumptions is violated. Notably, we prove a lower bound for a large class of locally unstable restless bandits, showing that local stability is particularly fundamental for exponential asymptotic optimality. Finally, we use simulations to demonstrate that the two-set policy outperforms previous policies on certain RB problems and performs competitively overall.

Achieving Exponential Asymptotic Optimality in Average-Reward Restless Bandits without Global Attractor Assumption

TL;DR

A lower bound for a large class of locally unstable restless bandits is proved, showing that local stability is particularly fundamental for exponential asymptotic optimality.

Abstract

We consider the infinite-horizon average-reward restless bandit problem. We propose a novel \emph{two-set policy} that maintains two dynamic subsets of arms: one subset of arms has a nearly optimal state distribution and takes actions according to an Optimal Local Control routine; the other subset of arms is driven towards the optimal state distribution and gradually merged into the first subset. We show that our two-set policy is asymptotically optimal with an optimality gap for an -armed problem, under the mild assumptions of aperiodic-unichain, non-degeneracy, and local stability. Our policy is the first to achieve \emph{exponential asymptotic optimality} under the above set of easy-to-verify assumptions, whereas prior work either requires a strong \emph{global attractor} assumption or only achieves an optimality gap. We further discuss obstacles in weakening the assumptions by demonstrating examples where exponential asymptotic optimality is not achievable when any of the three assumptions is violated. Notably, we prove a lower bound for a large class of locally unstable restless bandits, showing that local stability is particularly fundamental for exponential asymptotic optimality. Finally, we use simulations to demonstrate that the two-set policy outperforms previous policies on certain RB problems and performs competitively overall.
Paper Structure (45 sections, 32 theorems, 228 equations, 4 figures, 1 algorithm)

This paper contains 45 sections, 32 theorems, 228 equations, 4 figures, 1 algorithm.

Table of Contents

  1. Introduction
  2. Other related work.
  3. Problem setup
  4. Two-Set Policy
  5. LP relaxation
  6. Overview of our main policy
  7. Subroutine 1: Unconstrained Optimal Control
  8. Subroutine 2: Optimal Local Control (Local LP-Priority)
  9. When is the condition in \ref{['eq:lp-priority-condition-1']} true?
  10. Constructing the Two-Set Policy
  11. Main results
  12. Optimality guarantee on Two-Set Policy
  13. Discussion on the necessities of the assumptions
  14. When \ref{['assump:aperiodic-unichain']} (aperiodic unichain) or \ref{['assump:non-degeneracy']} (non-degeneracy) is violated.
  15. When \ref{['assump:local-stability']} is violated.
  16. ...and 30 more sections

Key Result

Theorem 1

Suppose each arm is unichain, aperiodic, non-degenerate, and locally stable. Then the two-set policy $\pi$ satisfies where $R^\textup{rel}$ is an upper bound on the optimal long-run average reward, and $R(\pi, \bm{S}_0)$ is the long-run average reward under the policy $\pi$ given the vector of initial states $\bm{S}_0$.

Figures (4)

  • Figure 1: RB instances satisfying Assumptions \ref{['assump:aperiodic-unichain']}, \ref{['assump:non-degeneracy']}, and \ref{['assump:local-stability']}, but violate UGAP.
  • Figure 2: First random instance sampled from uniform distribution.
  • Figure 3: Total optimality gap ratios of two additional uniformly-generated random instances.
  • Figure 4: The flowchart for the proof of \ref{['thm:two-set:achievability']}. We omit \ref{['lem:two-set:subroutine-conform']} from the flowchart for readability since it is used in the proofs of all other lemmas and the theorem.

Theorems & Definitions (58)

  • Theorem : Informal version of Theorem \ref{['thm:two-set:achievability']}
  • Theorem : Informal version of Theorem \ref{['thm:instability-lower-bound']}
  • Remark 1
  • Lemma 1
  • Lemma 2: Feasibility-ensuring pair
  • Definition 1: $\epsilon_N^{\text{rd}}$-maximal feasible set
  • Theorem 1
  • Remark 2
  • Definition 2: Regular unstable RBs
  • Theorem 2
  • ...and 48 more