Tracking and Assigning Jobs to a Markov Machine

TL;DR

This work studies tracking a two-state Markov machine in a time-slotted setting with a sampler and an FCFS server, aiming to minimize the sum of the age-of-incorrect-information penalty and a job-drop penalty. It casts the problem as an average-cost Markov decision process on states $s=(v,b)$ with actions $a\in\{0,1\}$ and cost $C(s,a)=Sv+aqp$, proving the existence of a stationary threshold policy with a threshold $v_{th}^*$ and deriving necessary and sufficient conditions for threshold optimality, along with a method to compute $v_{th}^*$ without bounding the state space. The analysis includes continuous-age extensions and numerical results detailing how $v_{th}^*$ and the average cost respond to parameters $p$, $S$, $q$, and $q_1$, highlighting monotonicity properties and the trade-offs between sampling effort and drop penalties. The results provide a principled, implementable policy structure for AoI-aware tracking in coupled tracking-and-control systems and advance understanding of sampling in Markovian, time-slotted environments.

Abstract

We consider a time-slotted communication system with a machine, a cloud server, and a sampler. Job requests from the users are queued on the server to be completed by the machine. The machine has two states, namely, a busy state and a free state. The server can assign a job to the machine in a first-in-first-served manner. If the machine is free, it completes the job request from the server; otherwise, it drops the request. Upon dropping a job request, the server is penalized. When the machine is in the free state, the machine can get into the busy state with an internal job. When the server does not assign a job request to the machine, the state of the machine evolves as a symmetric Markov chain. If the machine successfully accepts the job request from the server, the state of the machine goes to the busy state and follows a different dynamics compared to the dynamics when the machine goes to the busy state due to an internal job. The sampler samples the state of the machine and sends it to the server via an error-free channel. Thus, the server can estimate the state of the machine, upon receiving an update from the source. If the machine is in the free state but the estimated state at the server is busy, the sampler pays a cost. We incorporate the concept of the age of incorrect information to model the cost of the sampler. We aim to find an optimal sampling policy such that the cost of the sampler plus the penalty on the machine gets minimized. We formulate this problem in a Markov decision process framework and find how an optimal policy changes with several associated parameters. We show that a threshold policy is optimal for this problem. We show a necessary and sufficient condition for a threshold policy to be optimal. Finally, we find the optimal threshold without bounding the state space.

Figures (9)

• Figure 1: System model: A machine has two states, namely busy and free. A sampler samples the state of the machine and sends it to the server. Job requests from users are queued on the server to be completed by the machine. The server assigns jobs to the machine in a first-in-first-served manner.
• Figure 2: Pictorial representation of the dynamics of the Markov machine and the age of incorrect information penalty. The cyan boxes represent the busy state of the Markov machine when it is not serving any job request from the server, the green boxes represent the free state of the Markov machine, and the orange boxes represent the busy state of the Markov machine when it is serving a job request from the server. The green arrows represent the sampling instances, the red arrows represent an assignment of a job to the machine and the machine accepts that job, and the blue arrow represents an assignment of a job to the server, however, it does not accept the job as the machine has transitioned into the busy (internally) state, and thus pays the penalty $p$. We use $S=2$, for the evolution of age of incorrect information.
• Figure 3: Dynamics of the states of the Markov machine. The cyan circle corresponds to the busy state when the machine is not serving any job request from the server, the orange circle corresponds to the busy state when the machine is serving a job request, and the green circle corresponds to the free state. The blue arrow corresponds to action $a=0$, while the red arrow corresponds to action $a=1$. Here, we assume that $q'=1-q$ and $q_{1}'=1-q_{1}$.
• Figure 4: A pictorial representation of $M^{th}$. Here, we assume that $q'=1-q$ and $q_{1}'=1-q_{1}$.
• Figure 5: We compare the optimal threshold, a lower and an upper bound of optimal threshold for different $p$.

Theorems & Definitions (13)

• Lemma 1
• Corollary 1
• Theorem 1
• Remark 1
• Lemma 2
• Lemma 3
• Theorem 2
• Theorem 3
• Theorem 4
• Theorem 5