Which Sensor to Observe? Timely Tracking of a Joint Markov Source with Model Predictive Control

TL;DR

This work addresses remote tracking of a joint Markov process observed through multiple sensors with partial observations and an erasure channel, aiming to minimize mean AoII ($MAoII$). It casts the problem as a belief-MDP over the joint state and AoII, and solves it with two MPC-based policies: MPC-WTC (no terminal cost) and RL-MPC (learning-based terminal costs). Across two scenarios, the MPC approaches outperform baseline sensing schedules, with RL-MPC matching MPC-WTC performance while reducing offline complexity. The results demonstrate scalable, information-aware sensor selection for timely, accurate monitoring in networked sensing systems.

Abstract

In this paper, we investigate the problem of remote estimation of a discrete-time joint Markov process using multiple sensors. Each sensor observes a different component of the joint Markov process, and in each time slot, the monitor obtains a partial state value by sending a pull request to one of the sensors. The monitor chooses the sequence of sensors to observe with the goal of minimizing the mean of age of incorrect information (MAoII) by using the partial state observations obtained, which have different freshness levels. For instance, a monitor may be interested in tracking the location of an object by obtaining observations from two sensors, which observe the $x$ and $y$ coordinates of the object separately, in different time slots. The monitor, then, needs to decide which coordinate to observe in the next time slot given the history. In addition to this partial observability of the state of Markov process, there is an erasure channel with a fixed one-slot delay between each sensor and the monitor. First, we obtain a sufficient statistic, namely the \emph{belief}, representing the joint distribution of the age of incorrect information (AoII) and the current state of the observed process by using the history of all pull requests and observations. Then, we formulate the problem with a continuous state-space Markov decision problem (MDP), namely belief MDP. To solve the problem, we propose two model predictive control (MPC) methods, namely MPC without terminal costs (MPC-WTC) and reinforcement learning MPC (RL-MPC), that have different advantages in implementation.

Figures (5)

• Figure 1: The illustration of the system model. The processes that sensors observe are individually non-Markov, but their joint process is Markov with known transition probabilities. State space of the joint process $x(t)$ is ordered as $\{(a,\alpha),(b,\alpha),(a,\beta),(b,\beta)\}$ and its order is denoted with $\overline{x}_t$.
• Figure 2: Evolution of the belief from an initial belief $b_0$ for the process in Fig. \ref{['fig:sys']}. The first row of the transition matrix $\bm{P}$ is selected as $[0.6,0.1,0.1,0.2]$, and the remaining rows are circular-shifted one version of the previous row. After the first time step, the action $a_1=1$ is chosen, and for each possible observation $o_2\in\mathcal{O}(1)$, corresponding $\hat{b}_1$, $b_2$ and the probability of the observation is illustrated. For each case, the state with the maximum a-posteriori probability is highlighted in blue.
• Figure 3: An $\ell$-step look-ahead table that starts a belief state and explores all possible belief states on $\ell$ steps.
• Figure 4: Comparison of methods for varying $\rho_s$.
• Figure 5: Comparison of methods for different grid sizes when $\rho_s=0.8$.