Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Monitoring State Transitions in Markovian Systems with Sampling Cost

Kumar Saurav, Ness B. Shroff, Yingbin Liang

TL;DR

A projected stochastic gradient descent (PSGD)-based learning variant of the greedy policy is proposed, which achieves a favorable predict-query tradeoff with improved computational efficiency compared to OPT.

Abstract

We consider a node-monitor pair, where the node's state varies with time. The monitor needs to track the node's state at all times; however, there is a fixed cost for each state query. So the monitor may instead predict the state using time-series forecasting methods, including time-series foundation models (TSFMs), and query only when prediction uncertainty is high. Since query decisions influence prediction accuracy, determining when to query is nontrivial. A natural approach is a greedy policy that predicts when the expected prediction loss is below the query cost and queries otherwise. We analyze this policy in a Markovian setting, where the optimal (OPT) strategy is a state-dependent threshold policy minimizing the time-averaged sum of query cost and prediction losses. We show that, in general, the greedy policy is suboptimal and can have an unbounded competitive ratio, but under common conditions such as identically distributed transition probabilities, it performs close to OPT. For the case of unknown transition probabilities, we further propose a projected stochastic gradient descent (PSGD)-based learning variant of the greedy policy, which achieves a favorable predict-query tradeoff with improved computational efficiency compared to OPT.

Monitoring State Transitions in Markovian Systems with Sampling Cost

TL;DR

A projected stochastic gradient descent (PSGD)-based learning variant of the greedy policy is proposed, which achieves a favorable predict-query tradeoff with improved computational efficiency compared to OPT.

Abstract

We consider a node-monitor pair, where the node's state varies with time. The monitor needs to track the node's state at all times; however, there is a fixed cost for each state query. So the monitor may instead predict the state using time-series forecasting methods, including time-series foundation models (TSFMs), and query only when prediction uncertainty is high. Since query decisions influence prediction accuracy, determining when to query is nontrivial. A natural approach is a greedy policy that predicts when the expected prediction loss is below the query cost and queries otherwise. We analyze this policy in a Markovian setting, where the optimal (OPT) strategy is a state-dependent threshold policy minimizing the time-averaged sum of query cost and prediction losses. We show that, in general, the greedy policy is suboptimal and can have an unbounded competitive ratio, but under common conditions such as identically distributed transition probabilities, it performs close to OPT. For the case of unknown transition probabilities, we further propose a projected stochastic gradient descent (PSGD)-based learning variant of the greedy policy, which achieves a favorable predict-query tradeoff with improved computational efficiency compared to OPT.
Paper Structure (6 sections, 6 theorems, 3 equations, 4 figures, 4 algorithms)

This paper contains 6 sections, 6 theorems, 3 equations, 4 figures, 4 algorithms.

Table of Contents

  1. Introduction
  2. System Model
  3. Case $1$: Transition Probabilities are Known
  4. Case $2$: Transition Probabilities are Unknown
  5. Numerical Results
  6. Conclusion

Key Result

Lemma 1

Let $n_t=t-\tau(t)$, i.e. the time since last query until the current time $t$. Also, let the queried state at time $\tau(t)$ be $s_i$. At time t, the optimal prediction strategy is to pick state where $\ell_{jk}$ is the loss when the state is $s_j$ while the prediction is $s_k$. Also, the optimal (expected) prediction loss is $\sum_{s_j\in\mathcal{S}} p_{ij}^{(n_t)}\ell_{jk^\star}$.

Figures (4)

  • Figure 1: A neural network representation of the state transition process. This graph can be used to compute the error in state predictions, as well as the gradient of the loss w.r.t. probability estimates $\hat{p}_{ij}$.
  • Figure 2: Performance of policies w.r.t. query cost $c$ in a recurrent Markov chain.
  • Figure 3: Performance of uniform sampling policy w.r.t. sampling interval, and prediction strategy.
  • Figure 4: For randomly generated matrices $\mathcal{P}$, the performance of greedy policy $\pi^g$ is close to the optimal policy $\pi^\star$.

Theorems & Definitions (12)

  • Remark 1
  • Remark 2
  • Lemma 1
  • Theorem 1
  • Lemma 2
  • Lemma 3
  • Theorem 2
  • Remark 3
  • Remark 4
  • Remark 5
  • ...and 2 more