Revisiting Weighted Strategy for Non-stationary Parametric Bandits and MDPs
Jing Wang, Peng Zhao, Zhi-Hua Zhou
TL;DR
This work revisits the weighted forgetting approach for non-stationary parametric bandits and MDPs, identifying that historical analyses relied on problem-specific, complex weighted tools. It introduces a refined analysis framework that uses a single $V_{t-1}^{-1}$-norm for both bias and variance parts, along with a weighted potential lemma, enabling simpler, more efficient weight-based algorithms. The authors develop LB-WeightUCB, GLB-WeightUCB, and SCB-WeightUCB, achieving improved dynamic regret bounds (e.g., $\widetilde{\mathcal{O}}(d^{3/4} P_T^{1/4} T^{3/4})$ for LB under drift and analogous rates for GLB/SCB), and extend these ideas to non-stationary Linear Mixture MDP and MNL Mixture MDP with the WeightUCRL/MNL-WeightUCRL frameworks. Experiments on drifting LB and GLB validate the theoretical gains and illustrate practical efficiency improvements over restart/adaptive methods. Overall, the refined framework renders weighted strategies competitive with the best restart-based methods in non-stationary settings and broadens applicability to key online RL models with function approximation.
Abstract
Non-stationary parametric bandits have attracted much attention recently. There are three principled ways to deal with non-stationarity, including sliding-window, weighted, and restart strategies. As many non-stationary environments exhibit gradual drifting patterns, the weighted strategy is commonly adopted in real-world applications. However, previous theoretical studies show that its analysis is more involved and the algorithms are either computationally less efficient or statistically suboptimal. This paper revisits the weighted strategy for non-stationary parametric bandits. In linear bandits (LB), we discover that this undesirable feature is due to an inadequate regret analysis, which results in an overly complex algorithm design. We propose a \emph{refined analysis framework}, which simplifies the derivation and, importantly, produces a simpler weight-based algorithm that is as efficient as window/restart-based algorithms while retaining the same regret as previous studies. Furthermore, our new framework can be used to improve regret bounds of other parametric bandits, including Generalized Linear Bandits (GLB) and Self-Concordant Bandits (SCB). For example, we develop a simple weighted GLB algorithm with an $\tilde{O}(k_μ^{5/4} c_μ^{-3/4} d^{3/4} P_T^{1/4}T^{3/4})$ regret, improving the $\tilde{O}(k_μ^{2} c_μ^{-1}d^{9/10} P_T^{1/5}T^{4/5})$ bound in prior work, where $k_μ$ and $c_μ$ characterize the reward model's nonlinearity, $P_T$ measures the non-stationarity, $d$ and $T$ denote the dimension and time horizon. Moreover, we extend our framework to non-stationary Markov Decision Processes (MDPs) with function approximation, focusing on Linear Mixture MDP and Multinomial Logit (MNL) Mixture MDP. For both classes, we propose algorithms based on the weighted strategy and establish dynamic regret guarantees using our analysis framework.