Autoregressive Bandits

TL;DR

A novel online learning setting, namely, Autoregressive Bandits (ARBs), in which the observed reward is governed by an autoregressive process of order k, whose parameters depend on the chosen action, and a new optimistic regret minimization algorithm, namely, AutoRegressive Upper Confidence Bound (AR-UCB), that suffers sublinear regret of order.

Abstract

Autoregressive processes naturally arise in a large variety of real-world scenarios, including stock markets, sales forecasting, weather prediction, advertising, and pricing. When facing a sequential decision-making problem in such a context, the temporal dependence between consecutive observations should be properly accounted for guaranteeing convergence to the optimal policy. In this work, we propose a novel online learning setting, namely, Autoregressive Bandits (ARBs), in which the observed reward is governed by an autoregressive process of order $k$, whose parameters depend on the chosen action. We show that, under mild assumptions on the reward process, the optimal policy can be conveniently computed. Then, we devise a new optimistic regret minimization algorithm, namely, AutoRegressive Upper Confidence Bound (AR-UCB), that suffers sublinear regret of order $\widetilde{\mathcal{O}} \left( \frac{(k+1)^{3/2}\sqrt{nT}}{(1-Γ)^2}\right)$, where $T$ is the optimization horizon, $n$ is the number of actions, and $Γ< 1$ is a stability index of the process. Finally, we empirically validate our algorithm, illustrating its advantages w.r.t. bandit baselines and its robustness to misspecification of key parameters.

Figures (8)

• Figure 1: Settings and cumulative regret of AR-UCB and multiple baselines (100 runs, mean $\pm$ std).
• Figure 2: Effect of the choice of parameter $\overline{m}$ on the AR-UCB cumulative regret ($100$ runs, mean $\pm$ std).
• Figure 3: Effect of the choice of parameter $\overline{k}$ on the AR-UCB cumulative regret ($100$ runs, mean $\pm$ std).
• Figure 4: An illustration of the effect of a negative $\gamma_1(a)$ over time.
• Figure 5: The effect of $\gamma_1(a)$ in the evolution of the state $x_t$, in the case of a non-negative one (in black), and a negative one (in red).

Theorems & Definitions (21)

• Theorem 1: Optimal Policy
• Lemma 1: Self-Normalized Concentration
• Lemma 2: Policy Regret Decomposition
• Lemma 3: External-to-Policy Regret Bound
• Theorem 2
• Remark 1: Comparison with MDPs
• Theorem 2: Optimal Policy
• proof
• Lemma 3: Self-Normalized Concentration
• proof