Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Stochastic contextual bandits with graph feedback: from independence number to MAS number

Yuxiao Wen, Yanjun Han, Zhengyuan Zhou

TL;DR

With many contexts, the results show that the MAS number essentially characterizes the statistical complexity for contextual bandits, as opposed to the independence number in multi-armed bandits.

Abstract

We consider contextual bandits with graph feedback, a class of interactive learning problems with richer structures than vanilla contextual bandits, where taking an action reveals the rewards for all neighboring actions in the feedback graph under all contexts. Unlike the multi-armed bandits setting where a growing literature has painted a near-complete understanding of graph feedback, much remains unexplored in the contextual bandits counterpart. In this paper, we make inroads into this inquiry by establishing a regret lower bound $Ω(\sqrt{β_M(G) T})$, where $M$ is the number of contexts, $G$ is the feedback graph, and $β_M(G)$ is our proposed graph-theoretic quantity that characterizes the fundamental learning limit for this class of problems. Interestingly, $β_M(G)$ interpolates between $α(G)$ (the independence number of the graph) and $\mathsf{m}(G)$ (the maximum acyclic subgraph (MAS) number of the graph) as the number of contexts $M$ varies. We also provide algorithms that achieve near-optimal regret for important classes of context sequences and/or feedback graphs, such as transitively closed graphs that find applications in auctions and inventory control. In particular, with many contexts, our results show that the MAS number essentially characterizes the statistical complexity for contextual bandits, as opposed to the independence number in multi-armed bandits.

Stochastic contextual bandits with graph feedback: from independence number to MAS number

TL;DR

With many contexts, the results show that the MAS number essentially characterizes the statistical complexity for contextual bandits, as opposed to the independence number in multi-armed bandits.

Abstract

We consider contextual bandits with graph feedback, a class of interactive learning problems with richer structures than vanilla contextual bandits, where taking an action reveals the rewards for all neighboring actions in the feedback graph under all contexts. Unlike the multi-armed bandits setting where a growing literature has painted a near-complete understanding of graph feedback, much remains unexplored in the contextual bandits counterpart. In this paper, we make inroads into this inquiry by establishing a regret lower bound , where is the number of contexts, is the feedback graph, and is our proposed graph-theoretic quantity that characterizes the fundamental learning limit for this class of problems. Interestingly, interpolates between (the independence number of the graph) and (the maximum acyclic subgraph (MAS) number of the graph) as the number of contexts varies. We also provide algorithms that achieve near-optimal regret for important classes of context sequences and/or feedback graphs, such as transitively closed graphs that find applications in auctions and inventory control. In particular, with many contexts, our results show that the MAS number essentially characterizes the statistical complexity for contextual bandits, as opposed to the independence number in multi-armed bandits.
Paper Structure (29 sections, 14 theorems, 41 equations, 2 algorithms)

This paper contains 29 sections, 14 theorems, 41 equations, 2 algorithms.

Table of Contents

  1. Introduction
  2. Notations
  3. Our results
  4. Related work
  5. Hard instance and the regret lower bound
  6. Algorithms achieving the regret upper bounds
  7. Two sequential games on graphs
  8. Learning under self-avoiding contexts
  9. Learning under general contexts
  10. Discussions
  11. Weakly observable feedback graphs
  12. Incomplete cross-learning
  13. Weakly observable feedback graphs on actions
  14. Strongly observable feedback graphs on actions
  15. Gap between upper and lower bounds
  16. ...and 14 more sections

Key Result

Theorem 1.1

For $T \ge \beta_M(G)^3$, it holds that $\mathsf{R}_T^\star(G,M) = \Omega(\sqrt{\beta_M(G)T})$, where the graph-theoretic quantity $\beta_M(G)$ is given by and $I_i\not\rightarrow I_j$ means that $u\not\rightarrow v$ whenever $u\in I_i$ and $v\in I_j$.

Theorems & Definitions (17)

  • Theorem 1.1: Minimax lower bound
  • Corollary 1: Tightness of MAS number
  • Theorem 1.2: Upper bound for self-avoiding contexts
  • Theorem 1.3: Upper bound for general contexts
  • Corollary 2: Upper bound for transtively closed or undirected feedback
  • Definition 1: Sequential game I
  • Lemma 1: Minimax value of sequential game I
  • Definition 2: Sequential game II
  • Lemma 2: Minimax value of sequential game II
  • Theorem 3.1: Regret upper bound of Algorithm \ref{['alg:arm_elim_SA']}
  • ...and 7 more