The Law of Large Numbers and CLT for Non-stationary Markov Jump Processes Exhibiting Time-of-Day Effects

TL;DR

This work addresses non-stationary, time-inhomogeneous Markov jump processes with rich reward structures by proving a general LLN and CLT for cumulative rewards $R(t)$. The authors develop a martingale representation via a bounded non-stationary Poisson–type equation, and they establish both a non-periodic CLT and a specialized periodic CLT, with extensions to resetting dynamics. They derive integral and backward-ODE formulations for $\mathbb{E}R(t)$ and $\mathrm{Var}R(t)$, and discuss scalable computational strategies (ODE-based and Monte Carlo) as well as a periodic spectral approach that yields t-independent centering and variance. The numerical study across Prendiville, single-server, and multi-server queues demonstrates the practical accuracy and efficiency of the CLT approximations for large horizons, highlighting their utility in service-operations planning under time-of-day effects.

Abstract

In this paper, we develop a general law of large numbers and central limit theorem for cumulative reward processes associated with finite state Markov jump processes with non-stationary transition rates. Such models commonly arise in service operations and manufacturing applications in which time-of-day, day-of-week, and secular effects are of first-order importance in predicting system behavior. Our theorems allow for non-stationary reward environments that continuously accumulate reward, while also including contributions from non-stationary lump-sum rewards of random size that are collected at either jump times of the underlying process, jump times of a Poisson process modulated by the underlying process, or scheduled deterministic times. As part of our development, we also obtain a new central limit theorem for the special case in which the jump process transition rates and reward structure are periodic (as may occur over a weekly time interval), as well as for jump process models with resetting. We include a simulation study illustrating the quality of our CLT approximations for several non-stationary stochastic models.

Figures (4)

• Figure 1: Error convergence plots for varying orders of numerical ODE methods for solving the ODEs for $\mathbb{E} R(1)$ in the Prendiville model. Dashed lines indicate linear least-squares regression lines on the log-log scale; the slopes reported in the legend, indicate the estimated order of convergence. The vertical axis plots the relative numerical error on log scale.
• Figure 2: Comparison of Runge-Kutta methods for solving the Prendiville ODEs for $\mathbb{E}\,R(1)$ with and without adding the discontinuities of $\tilde{r}$ to the collection of $t_i$'s. The label (DA) indicates a discontinuity-aware method, whereas the label (naive) indicates a discontinuity-unaware method.
• Figure 3: Rates $\lambda(t)$ and $\mu(t)$ for the $M_t / M_t / 1 / 30$ queue.
• Figure 4: Arrival rate $\lambda(t)$ and total service rate for the call center multi-server queue. Each time unit corresponds to one 8-hour shift, so that e.g. $t=3$ gives the arrival and total service rate at the end of the first 24 hours.

Theorems & Definitions (39)

• Proposition 1
• Remark 1
• proof : Proof of Proposition \ref{['prop:mixing']}
• Remark 2
• Theorem 1
• proof
• Proposition 2
• proof
• Theorem 2
• proof