Learning to Admit Optimally in an $M/M/k/k+N$ Queueing System with Unknown Service Rate

TL;DR

This paper addresses admission control in an $M/M/k/k+N$ queue when the service rate $\mu$ is unknown, and decisions are made only at arrivals based on observed interarrival times and occupancy. It introduces a maximum-likelihood estimation approach combined with certainty-equivalent control and forced exploration to learn $\mu$ and select between admitting or blocking; the resulting policy $\pi_{Alg1}$ is shown to be asymptotically optimal and to enjoy finite-time regret guarantees. The analysis covers both no-waiting ($N=0$) and finite-waiting ($N>0$) scenarios, employing martingale techniques, Doob's decomposition, and coupling arguments to derive constant regret in favorable regimes and logarithmic (or polylogarithmic) regret in challenging ones. Simulation results corroborate the theory and illustrate robustness to sampling choices and comparisons with RL baselines, highlighting the practical relevance for sizing and pricing problems in networks and service systems.

Abstract

Motivated by applications of the Erlang-B blocking model and the extended $M/M/k/k+N$ model that allows for some queueing, beyond communication networks to sizing and pricing in production, messaging, and app-based parking systems, we study admission control for such systems with unknown service rate. In our model, a dispatcher either admits every arrival into the system (when there is room) or blocks it. Every served job yields a fixed reward but incurs a per unit time holding cost which includes the waiting time in the queue to get service if there is any. We aim to design a dispatching policy that maximizes the long-term average reward by observing arrival times and system state at arrivals, a realistic decision-event driven sampling of such systems. The dispatcher observes neither service times nor departure epochs, which excludes the use of reward-based reinforcement learning approaches. We develop our learning-based dispatch scheme as a parametric learning problem a'la self-tuning adaptive control. In our problem, certainty equivalent control switches between always admit if room (explore infinitely often), and never admit (terminate learning), so at judiciously chosen times we avoid the never admit recommendation. We prove that our proposed policy asymptotically converges to the optimal policy and present finite-time regret guarantees. The extreme contrast in the control policies shows up in our regret bounds for different parameter regimes: constant in one versus logarithmic in another.

Figures (8)

• Figure 1: Comparison of regret performance of \ref{['alg1']} for different functions $f(n)$ in a 5 server system with $\lambda=5$ and $c/R=1.3$. The shaded region indicate the $\pm \sigma$ area of mean regret.
• Figure 2: Variations of regret for different service rates in a 5 server system with $\lambda=5$, $c/R=1.3$, $\epsilon=0.4$, $\frac{1}{1-\epsilon}=5/3$, and $f(n)=\exp\left(n^{1-\epsilon}\right)$ following \ref{['alg1']}.
• Figure 3: Comparison of regret performance of \ref{['alg1']} against Algorithm 2 in a 5 server system with $\lambda=5$, $c/R=1.3$, $\epsilon=0.4$, $\frac{1}{1-\epsilon}=5/3$, and $f(n)=\exp\left(n^{1-\epsilon}\right)$.
• Figure 4: Comparison of regret performance of \ref{['alg1']} against RL algorithms in a 5 server system with $\lambda=5$, $c/R=1.3$, $\varepsilon=0.2$, and $f(n)=\exp\left(n^{1-\epsilon_n}\right)$.
• Figure 5: Comparison of regret performance of \ref{['alg1']} for different functions $f(n)$ in a 5 server system with $\lambda=5$, $c/R=1.3$, and $\epsilon=\varepsilon=0.55$.

Theorems & Definitions (26)

• Remark 1
• Definition 1
• Lemma 1
• Lemma 2
• Proposition 1
• Lemma 3
• Theorem 1
• Lemma 4
• Theorem 2
• Lemma 5