Learning to Route and Schedule LLMs from User Retrials via Contextual Queueing Bandits

TL;DR

This work tackles the problem of efficiently routing and scheduling queries for multiple LLMs under user retrials and limited explicit feedback. It introduces CQB-MNL, a contextual queueing bandit framework with multinomial logit feedback, and the anytime ACQB algorithm that combines decaying uniform exploration with Thompson sampling to learn routing and scheduling policies while ensuring queue stability. The paper provides regret guarantees, including a queue-length regret that decays as $R_t = \widetilde{\mathcal{O}}(t^{-1/4})$ and a cumulative regret of $\widetilde{\mathcal{O}}(d^{3/2}\sqrt{t})$, and validates the approach on synthetic and real-world datasets (SPROUT, EmbedLLM, RouterBench). It also proposes two practical extensions: disjoint parameterization to capture LLM heterogeneity and a contrastive-learning-based, utility-aligned embedding scheme (ACQB-CL) to improve routing utilities. Overall, the framework provides an anytime, scalable solution for real-time LLM routing/scheduling with implicit feedback, improving throughput and user experience in high-demand settings.

Abstract

Explosive demands for LLMs often cause user queries to accumulate in server queues, requiring efficient routing (query-LLM matching) and scheduling (query prioritization) mechanisms. Several online algorithms are being deployed, but they overlook the following two key challenges inherent to conversational LLM services: (1) unsatisfied users may retry queries, increasing the server backlog, and (2) requests for ``explicit" feedback, such as ratings, degrade user experiences. In this paper, we develop a joint routing and scheduling algorithm that leverages ``implicit" feedback inferred from user retrial behaviors. The key idea is to propose and study the framework of contextual queueing bandits with multinomial logit feedback (CQB-MNL). CQB-MNL models query retrials, as well as context-based learning for user preferences over LLMs. Our algorithm, anytime CQB (ACQB), achieves efficient learning while maintaining queue stability by combining Thompson sampling with forced exploration at a decaying rate. We show that ACQB simultaneously achieves a cumulative regret of $\widetilde{\mathcal{O}}(\sqrt{t})$ for routing and a queue length regret of $\widetilde{\mathcal{O}}(t^{-1/4})$ for any large $t$. For experiments, we refine query embeddings via contrastive learning while adopting a disjoint parameter model to learn LLM-specific parameters. Experiments on SPROUT, EmbedLLM, and RouterBench datasets confirm that both algorithms consistently outperform baselines.

Figures (7)

• Figure 1: Illustration demonstrating retrial and departure dynamics. ($K=1$): ① The agent schedules a query (Query 1). ② An assortment of size $K=1$ is assigned. ③ The user is dissatisfied with the response, which ④ triggers a retrial. ($K=2$): ⑤ The agent schedules a query (Query 3). ⑥ An assortment of size $K=2$ is assigned. ⑦ The user selects one of the responses (satisfaction), and consequently, ⑧ the query departs the queue.
• Figure 2: Queue length and cumulative regret on synthetic data with $\lambda=0.7$, $\epsilon=0.03$, and $N=5$.
• Figure 3: Queue length and cumulative regret on the SPROUT-o3mini dataset across various arrival rates $\lambda$. ACQB-CL consistently achieves the lowest regret across all settings, demonstrating the effectiveness of our utility-based contrastive learning.
• Figure 4: Queue length and cumulative regret on synthetic data with $\lambda=0.7$, $\epsilon=0.03$, and varying $N \in \{3,5,10\}$.
• Figure 5: Queue length and cumulative regret on synthetic data with $\lambda=0.7$, $N=5$, and varying $\epsilon\in\{0.05,0.03,0.01\}$.

Theorems & Definitions (42)

• Theorem 5
• Theorem 6
• Remark 7
• Lemma 8
• Lemma 9
• Proposition 10: Per-round regret
• Lemma 11
• Lemma 12
• Proposition 13
• Proposition 14