Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Online Continuous Hyperparameter Optimization for Generalized Linear Contextual Bandits

Yue Kang, Cho-Jui Hsieh, Thomas C. M. Lee

TL;DR

This work tackles the practical challenge of tuning hyperparameters in generalized linear contextual bandits in real time, where offline methods like cross-validation are infeasible. It introduces Continuous Dynamic Tuning (CDT), a two-layer framework where the top layer treats hyperparameter configurations as arms in a non-stationary Lipschitz continuum-armed bandit and the bottom layer runs a contextual bandit using those hyperparameters. The top layer employs Zooming TS with Restarts within a Bandit-over-Bandit scheme to adaptively zoom into promising regions while handling piecewise stationarity, achieving sublinear dynamic regret under switching environments; in stationary settings, it attains near-optimal rates. Empirically, CDT outperforms existing hyperparameter tuning approaches across multiple GLB algorithms on synthetic data and real datasets (Movielens and Yahoo), with stable running times, demonstrating its practicality for online decision-making systems. Overall, the paper provides both theoretical guarantees and strong empirical evidence for online continuous hyperparameter optimization in contextual bandits, advancing the field toward scalable model selection in online learning.

Abstract

In stochastic contextual bandits, an agent sequentially makes actions from a time-dependent action set based on past experience to minimize the cumulative regret. Like many other machine learning algorithms, the performance of bandits heavily depends on the values of hyperparameters, and theoretically derived parameter values may lead to unsatisfactory results in practice. Moreover, it is infeasible to use offline tuning methods like cross-validation to choose hyperparameters under the bandit environment, as the decisions should be made in real-time. To address this challenge, we propose the first online continuous hyperparameter tuning framework for contextual bandits to learn the optimal parameter configuration in practice within a search space on the fly. Specifically, we use a double-layer bandit framework named CDT (Continuous Dynamic Tuning) and formulate the hyperparameter optimization as a non-stationary continuum-armed bandit, where each arm represents a combination of hyperparameters, and the corresponding reward is the algorithmic result. For the top layer, we propose the Zooming TS algorithm that utilizes Thompson Sampling (TS) for exploration and a restart technique to get around the \textit{switching} environment. The proposed CDT framework can be easily utilized to tune contextual bandit algorithms without any pre-specified candidate set for multiple hyperparameters. We further show that it could achieve a sublinear regret in theory and performs consistently better than all existing methods on both synthetic and real datasets.

Online Continuous Hyperparameter Optimization for Generalized Linear Contextual Bandits

TL;DR

This work tackles the practical challenge of tuning hyperparameters in generalized linear contextual bandits in real time, where offline methods like cross-validation are infeasible. It introduces Continuous Dynamic Tuning (CDT), a two-layer framework where the top layer treats hyperparameter configurations as arms in a non-stationary Lipschitz continuum-armed bandit and the bottom layer runs a contextual bandit using those hyperparameters. The top layer employs Zooming TS with Restarts within a Bandit-over-Bandit scheme to adaptively zoom into promising regions while handling piecewise stationarity, achieving sublinear dynamic regret under switching environments; in stationary settings, it attains near-optimal rates. Empirically, CDT outperforms existing hyperparameter tuning approaches across multiple GLB algorithms on synthetic data and real datasets (Movielens and Yahoo), with stable running times, demonstrating its practicality for online decision-making systems. Overall, the paper provides both theoretical guarantees and strong empirical evidence for online continuous hyperparameter optimization in contextual bandits, advancing the field toward scalable model selection in online learning.

Abstract

In stochastic contextual bandits, an agent sequentially makes actions from a time-dependent action set based on past experience to minimize the cumulative regret. Like many other machine learning algorithms, the performance of bandits heavily depends on the values of hyperparameters, and theoretically derived parameter values may lead to unsatisfactory results in practice. Moreover, it is infeasible to use offline tuning methods like cross-validation to choose hyperparameters under the bandit environment, as the decisions should be made in real-time. To address this challenge, we propose the first online continuous hyperparameter tuning framework for contextual bandits to learn the optimal parameter configuration in practice within a search space on the fly. Specifically, we use a double-layer bandit framework named CDT (Continuous Dynamic Tuning) and formulate the hyperparameter optimization as a non-stationary continuum-armed bandit, where each arm represents a combination of hyperparameters, and the corresponding reward is the algorithmic result. For the top layer, we propose the Zooming TS algorithm that utilizes Thompson Sampling (TS) for exploration and a restart technique to get around the \textit{switching} environment. The proposed CDT framework can be easily utilized to tune contextual bandit algorithms without any pre-specified candidate set for multiple hyperparameters. We further show that it could achieve a sublinear regret in theory and performs consistently better than all existing methods on both synthetic and real datasets.
Paper Structure (30 sections, 12 theorems, 100 equations, 5 figures, 6 tables, 3 algorithms)

This paper contains 30 sections, 12 theorems, 100 equations, 5 figures, 6 tables, 3 algorithms.

Table of Contents

  1. Introduction
  2. Related Work
  3. Preliminaries
  4. Main Results
  5. Zooming TS Algorithm with Restarts
  6. Online Continuous Hyperparameter Optimization for Contextual Bandits
  7. Experimental Results
  8. Conclusion
  9. Limitations and future works:
  10. Supportive Experimental Details
  11. Simulations on the Optimal Hyperparameter Value in Grid Search
  12. Simulations to Validate the Lipschitzness of Hyperparameter Configuration
  13. Simulations for Algorithm \ref{['alg:zoomts']}
  14. Additional Details and Results for Section \ref{['sec:exp']}
  15. Baselines with A Large Candidate Set
  16. ...and 15 more sections

Key Result

Theorem 4.1

With $H = \Theta \left((T/c(T))^{(p_{z,*}+2)/(p_{z,*}+3)}]\right)$, the total regret of our Zooming TS algorithm with Restarts under the switching environment over time $T$ is bounded as when $c(T)>0$. In addition, if the environment is stationary (i.e. $c(T)=0,f_t=f,p_{z,t}=p_{z,*}\coloneqq p_z, \forall t \in [T]$), then by using $H=T$ (i.e. no restart), our Zooming TS algorithm could achieve th

Figures (5)

  • Figure 1: Illustration of the restarted strategy.
  • Figure 2: Cumulative regret curves of our CDT framework compared with existing hyperparameter selection methods under multiple (generalized) linear bandit algorithms on the simulations and Movielens dataset.
  • Figure 3: Average cumulative regret and its standard deviation of group mean reward for different hyperparameter values across all groups.
  • Figure 4: Cumulative regret plots of Zooming TS-R, Zooming and Oracle algorithms under the switching environment.
  • Figure 5: An illustration of Zooming TS algorithm with double restarts when $c(T)$ is agnostic.

Theorems & Definitions (15)

  • Theorem 4.1
  • Theorem 4.2
  • Remark B.1
  • Definition E.1
  • Lemma E.2
  • Lemma E.3
  • Lemma E.4
  • Lemma E.5
  • Theorem F.1
  • Lemma G.1: Proposition 1 in li2017provably
  • ...and 5 more