Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Stochastic Multi-Armed Bandits with Limited Control Variates

Arun Verma, Manjesh Kumar Hanawal, Arun Rajkumar

TL;DR

This work proposes UCB-LCV, an upper confidence bound (UCB) based algorithm that effectively combines the estimators obtained from rewards and control variates, and demonstrates that UCB-LCV outperforms existing bandit algorithms.

Abstract

Motivated by wireless networks where interference or channel state estimates provide partial insight into throughput, we study a variant of the classical stochastic multi-armed bandit problem in which the learner has limited access to auxiliary information. Recent work has shown that such auxiliary information, when available as control variates, can be used to get tighter confidence bounds, leading to lower regret. However, existing works assume that control variates are available in every round, which may not be realistic in several real-life scenarios. To address this, we propose UCB-LCV, an upper confidence bound (UCB) based algorithm that effectively combines the estimators obtained from rewards and control variates. When there is no control variate, UCB-LCV leads to a novel algorithm that we call UCB-NORMAL, outperforming its existing algorithms for the standard MAB setting with normally distributed rewards. Finally, we discuss variants of the proposed UCB-LCV that apply to general distributions and experimentally demonstrate that UCB-LCV outperforms existing bandit algorithms.

Stochastic Multi-Armed Bandits with Limited Control Variates

TL;DR

This work proposes UCB-LCV, an upper confidence bound (UCB) based algorithm that effectively combines the estimators obtained from rewards and control variates, and demonstrates that UCB-LCV outperforms existing bandit algorithms.

Abstract

Motivated by wireless networks where interference or channel state estimates provide partial insight into throughput, we study a variant of the classical stochastic multi-armed bandit problem in which the learner has limited access to auxiliary information. Recent work has shown that such auxiliary information, when available as control variates, can be used to get tighter confidence bounds, leading to lower regret. However, existing works assume that control variates are available in every round, which may not be realistic in several real-life scenarios. To address this, we propose UCB-LCV, an upper confidence bound (UCB) based algorithm that effectively combines the estimators obtained from rewards and control variates. When there is no control variate, UCB-LCV leads to a novel algorithm that we call UCB-NORMAL, outperforming its existing algorithms for the standard MAB setting with normally distributed rewards. Finally, we discuss variants of the proposed UCB-LCV that apply to general distributions and experimentally demonstrate that UCB-LCV outperforms existing bandit algorithms.
Paper Structure (15 sections, 4 theorems, 29 equations, 4 figures, 2 algorithms)

This paper contains 15 sections, 4 theorems, 29 equations, 4 figures, 2 algorithms.

Table of Contents

  1. Introduction
  2. Related Work
  3. Problem Setting
  4. Normally Distributed Rewards and CVs
  5. Mean Reward Estimator
  6. Algorithm: \ref{['alg:UCB-LCV']}
  7. Estimator with Multiple Control Variates
  8. Special Cases of MAB-LCV
  9. MAB-LCV problems having no CVs
  10. CVs for all rewards samples
  11. General Distribution
  12. Experiments
  13. Gaussian Distribution
  14. General Distribution
  15. Conclusion

Key Result

Proposition 1

Let $m=M_i(t)$, $n=N_i(t)$, and $s=S_i(t)$ for arm $i \in [K]$ at the beginning of round $t$. For any $\lambda_{t,i} \in (0,1)$, the mean reward estimator $\hat{\mu}_{s,i}$ has following properties:

Figures (4)

  • Figure 1: Variation in $\frac{\mathcal{V}_{T,N_i(T)}^{(2)}}{\mathcal{V}_{T,T}^{(2)}}$ with $S_i(T)$.
  • Figure 2: $\mathcal{V}_{T,T}^{(\alpha)}$ vs. $\log(T)$.
  • Figure 3: Regret comparison of \ref{['alg:UCB-LCV']} with existing bandit algorithms (Fig. \ref{['fig:3a']} and Fig. \ref{['fig:3b']}). Regret of \ref{['alg:UCB-LCV']} vs. the availability of control variate (Fig. \ref{['fig:3c']}). Regret of \ref{['alg:UCB-LCV']} vs. erroneous mean of control variate (Fig. \ref{['fig:3d']}). Since the confidence intervals are very small compared to the scale of the y-axis, they are not observable in the plots.
  • Figure 4: Regret comparison of different variants of \ref{['alg:UCB-LCV']} that are based on Jackknifing, Splitting, and Batching methods for MAB-LCV problems with arms having a non-Gaussian distribution of reward and associated CVs. In these experiments, \ref{['alg:UCB-LCV']} assumes a Gaussian distribution irrespective of the underlying true distribution.

Theorems & Definitions (9)

  • Proposition 1
  • proof
  • Lemma 1
  • proof
  • Lemma 2
  • proof
  • Proposition 2
  • Remark 1
  • Remark 2: Regret Analysis of \ref{['alg:UCB-LCV']}: