Queue Length Regret Bounds for Contextual Queueing Bandits

TL;DR

This work introduces contextual queueing bandits, a framework where jobs arrive with heterogeneous contextual features and the departure rate follows a logistic model with unknown server parameters, requiring simultaneous learning and scheduling. A novel coupling-based technique using policy-switching queues addresses queue-state misalignment, enabling a decomposition of queue-length regret into short-term and long-term effects. The authors propose two algorithms, CQB-$\varepsilon$ for stochastic contexts and CQB-Opt for adversarial contexts, achieving a decaying $\widetilde{O}(T^{-1/4})$ regret in the stochastic setting and $\mathcal{O}(\log^2 T)$ in the adversarial setting, both supported by experimental validation. The results advance understanding of learning-while-scheduling under context-rich arrivals and provide practical insights for designing stable, learning-enabled queueing systems with context-aware decision making.

Abstract

We introduce contextual queueing bandits, a new context-aware framework for scheduling while simultaneously learning unknown service rates. Individual jobs carry heterogeneous contextual features, based on which the agent chooses a job and matches it with a server to maximize the departure rate. The service/departure rate is governed by a logistic model of the contextual feature with an unknown server-specific parameter. To evaluate the performance of a policy, we consider queue length regret, defined as the difference in queue length between the policy and the optimal policy. The main challenge in the analysis is that the lists of remaining job features in the queue may differ under our policy versus the optimal policy for a given time step, since they may process jobs in different orders. To address this, we propose the idea of policy-switching queues equipped with a sophisticated coupling argument. This leads to a novel queue length regret decomposition framework, allowing us to understand the short-term effect of choosing a suboptimal job-server pair and its long-term effect on queue state differences. We show that our algorithm, CQB-$\varepsilon$, achieves a regret upper bound of $\widetilde{\mathcal{O}}(T^{-1/4})$. We also consider the setting of adversarially chosen contexts, for which our second algorithm, CQB-Opt, achieves a regret upper bound of $\mathcal{O}(\log^2 T)$. Lastly, we provide experimental results that validate our theoretical findings.

Figures (4)

• Figure 1: Illustration of the queueing processes under our policy $\pi$ and the optimal policy $\pi^*$ in contextual queueing bandits with three servers. Due to a suboptimal choice by our policy $\pi$ in round $t+1$, the queue states diverge in round $t+2$, where we call this queue state misalignment.
• Figure 2: Average queue length across algorithms and settings. (Left) CQB-$\boldsymbol{\varepsilon}$ and CQB-Opt versus a random policy, the optimal policy, and additional baselines. (Middle) and (right) performance of CQB-$\boldsymbol{\varepsilon}$ and CQB-Opt, respectively, for $\epsilon\in\{0.05,0.1,0.15\}$
• Figure 3: Average queue length across varying $K$ and $d$. (Top-left/right) CQB-$\boldsymbol{\varepsilon}$/CQB-Opt for $K\in\{3,5,7,9\}$. (Bottom-left/right) CQB-$\boldsymbol{\varepsilon}$/CQB-Opt for $d\in\{3,5,10,20\}$.
• Figure 4: Average queue length across algorithms and settings. (Left) shows the average queue length for MNIST. (Middle) and (Right) show the performance for Heart Disease and In-Vehicle Coupon Recommendation, respectively.

Theorems & Definitions (42)

• Lemma 4.1
• Lemma 4.2
• Lemma 5.1
• Lemma 5.2
• Lemma 5.3
• Lemma 5.4
• Theorem 5.5
• proof : Proof sketch
• Proposition 6.1
• Lemma 6.2