Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Decomposition-Coordination Method for Finite Horizon Bandit Problems

Michel de Lara, Benjamin Heymann, Jean-Philippe Chancelier

Abstract

Optimally solving a multi-armed bandit problem suffers the curse of dimensionality. Indeed, resorting to dynamic programming leads to an exponential growth of computing time, as the number of arms and the horizon increase. We introduce a decompositioncoordination heuristic, DeCo, that turns the initial problem into parallelly coordinated one-armed bandit problems. As a consequence, we obtain a computing time which is essentially linear in the number of arms. In addition, the decomposition provides a theoretical lower bound on the regret. For the two-armed bandit case, dynamic programming provides the exact solution, which is almost matched by the DeCo heuristic. Moreover, in numerical simulations with up to 100 rounds and 20 arms, DeCo outperforms classic algorithms (Thompson sampling and Kullback-Leibler upper-confidence bound) and almost matches the theoretical lower bound on the regret for 20 arms.

Decomposition-Coordination Method for Finite Horizon Bandit Problems

Abstract

Optimally solving a multi-armed bandit problem suffers the curse of dimensionality. Indeed, resorting to dynamic programming leads to an exponential growth of computing time, as the number of arms and the horizon increase. We introduce a decompositioncoordination heuristic, DeCo, that turns the initial problem into parallelly coordinated one-armed bandit problems. As a consequence, we obtain a computing time which is essentially linear in the number of arms. In addition, the decomposition provides a theoretical lower bound on the regret. For the two-armed bandit case, dynamic programming provides the exact solution, which is almost matched by the DeCo heuristic. Moreover, in numerical simulations with up to 100 rounds and 20 arms, DeCo outperforms classic algorithms (Thompson sampling and Kullback-Leibler upper-confidence bound) and almost matches the theoretical lower bound on the regret for 20 arms.

Paper Structure

This paper contains 21 sections, 6 theorems, 39 equations, 4 figures, 1 table.

Table of Contents

  1. Introduction
  2. Preliminaries
  3. The Bayesian multi-armed bandit problem
  4. Probabilistic model
  5. Decision model
  6. Information and admissible controls
  7. Random rewards
  8. State variable
  9. Optimality criteria in the Bayesian framework
  10. Arm Decomposition
  11. The value of information interpretation
  12. Characterization of the optimality gap
  13. Comparative analysis of admissible and relaxed solution performances
  14. Optimality gap estimate for DeCo
  15. Illustration
  16. ...and 6 more sections

Key Result

Proposition \oldthetheorem

We have the upper bound where we identify (by an abuse of notation) $V_0({({n^{\texttt{B}a}_{0}})_{a \in A}, ({n^{\texttt{G}a}_{0}})_{a \in A}})$ with the value $V_{0}({\pi_{0}})$ of problem eq:bandit_problem when the prior $\pi_{0} = \left\{\beta({n^{\texttt{B}a}_{0},n^{\texttt{G}a}_{0}})\right\}_{a\in A}$ , is a Beta d

Figures (4)

  • Figure 1: The decomposition coordination algorithm (DeCo)
  • Figure 2: Four simulations of trajectories generated with the controls obtained from DeCo for a uniform prior and 8 arms. In each of the four subfigures, we display the value of information $\delta$ (top), the expected reward $\ell$ (middle) and the index $I$ (bottom) obtained with DeCo. The green line corresponds to the value of the component $\mu_t$ of the multiplier at the end of the dual gradient procedure.
  • Figure 3: Expected Bayesian regret \ref{['eq:bayesianRegret']} for DeCo and a few benchmark policies (the lower the better) for 2, 5, 15 and 20 arms with uniform prior.
  • Figure 4: Expected Bayesian regret \ref{['eq:bayesianRegret']} for DeCo, Ts and Kl-Ucb with uniform prior, as functions of the number $\sharp {Arms}$ of arms. The (DeCo) lower bound Lb in \ref{['eq:lower_bound_bayesianRegret']} is also plotted and demonstrates that DeCo is close to the optimal solution when the number $\sharp {Arms}$ of arms is large enough.

Theorems & Definitions (6)

  • Proposition \oldthetheorem
  • Theorem \oldthetheorem
  • Lemma \oldthetheorem
  • Lemma \oldthetheorem
  • Lemma \oldthetheorem
  • Theorem \oldthetheorem