An Adaptive Method for Contextual Stochastic Multi-armed Bandits with Rewards Generated by a Linear Dynamical System

TL;DR

This work addresses online decision-making where rewards are generated by a Linear Gaussian Dynamical System (LGDS). It develops a matrix-based steady-state Kalman filter representation to predict rewards using a past-context window of length $s$, with $s$ chosen adaptively based on model uncertainty. The proposed Adaptive Recursive Least-Squares Exploration of System (ARES) combines an optimism-driven perturbation and an adaptive window-size mechanism to jointly control exploration and predictor complexity, yielding regret bounds that account for non-stationarity. Empirical results on synthetic LGDS tasks show that ARES and related approaches improve cumulative rewards over standard baselines, highlighting practical value for dynamic, context-rich bandit problems.

Abstract

Online decision-making can be formulated as the popular stochastic multi-armed bandit problem where a learner makes decisions (or takes actions) to maximize cumulative rewards collected from an unknown environment. This paper proposes to model a stochastic multi-armed bandit as an unknown linear Gaussian dynamical system, as many applications, such as bandits for dynamic pricing problems or hyperparameter selection for machine learning models, can benefit from this perspective. Following this approach, we can build a matrix representation of the system's steady-state Kalman filter that takes a set of previously collected observations from a time interval of length $s$ to predict the next reward that will be returned for each action. This paper proposes a solution in which the parameter $s$ is determined via an adaptive algorithm by analyzing the model uncertainty of the matrix representation. This algorithm helps the learner adaptively adjust its model size and its length of exploration based on the uncertainty of its environmental model. The effectiveness of the proposed scheme is demonstrated through extensive numerical studies, revealing that the proposed scheme is capable of increasing the rate of collected cumulative rewards.

Figures (5)

• Figure 1: The parameter $s$ is the sliding window size of contexts $\theta_{t-s}$ to $\theta_{t-1}$ that are used for predicting the reward $X_t$. For example, if $s = 3$, then at round $t$ the learner uses contexts $\theta_{t-3}$ to $\theta_{t-1}$ for predicting the reward $X_t$. The set $\mathscr{T}_a$ has all the rounds $t_i$ where action $a \in [k]$ is chosen and $N_a$ is the number of times action $a \in [k]$ that has been chosen by the learner. For example, if the learner choose action $a = 1$ at rounds $t_1$, $t_2$ and $t_3$, then set $\mathscr{T}_1= \{t_1,t_2,t_3\}$ and $N_1 = 3$.
• Figure 2: In step 1, the learner selects the window size $s_a$ for each action $a \in [k]$. As for step 2, the learner selects the action $a \in [k]$ that maximizes the quantity $\hat{G}_a^t\left(s_a\right)^\top\Theta_t + u_a$ with the chosen $s_a$.
• Figure 3: Plots on the left are median decrease/increase (positive percent/negative percent) in regret with respect to ARES's regret. The bottom and top part of the boxes are the first and third quantiles, respectively. Plots on the right are the median regret for each algorithm for each round.
• Figure 4: Plots on the left are average $u_a$ values for actions $a = 1,2$. Plots on the right are average chosen parameter $s$ values for actions $a = 1,2$.
• Figure 5: Plots on the left are average errors $\left\vert X_t - \hat{G}_a^t\left(s\right)^\top \Theta_t \right\vert$ for actions $a = 1$ (left plot in yellow) and $a = 2$ (right plot in blue).

Theorems & Definitions (27)

• Remark 1
• Remark 2
• Remark 3
• Remark 4
• Remark 5
• Theorem 1
• Lemma 1
• Theorem 2
• Corollary 1
• Theorem 3