Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

MDPs with a State Sensing Cost

Vansh Kapoor, Jayakrishnan Nair

TL;DR

This work addresses MDPs with an explicit cost for state sensing, formulating an augmented, countably infinite MDP \\mathcal{M}_k\\ that allows the agent to sense or remain blind at each step. It introduces SPI, a selective policy-improvement algorithm that greedily searches for improving blind-action sequences and leverages a myopic sensing value to guide updates, achieving near-optimal performance in practice. The authors establish provable guarantees through a threshold on the sensing cost for always-sense optimality and a truncation framework that yields computable lower bounds and suboptimality measures, enabling tractable planning for large or continuous state spaces. Empirically, SPI outperforms state-of-the-art POMDP solvers and the ATM heuristic across diverse domains (e.g., ICU-Sepsis, Frozen Lake, Taxi) with reasonable compute times, and extensions to non-uniform costs and continuous spaces are discussed, highlighting practical impact for healthcare, robotics, and sensor networks.

Abstract

In many practical sequential decision-making problems, tracking the state of the environment incurs a sensing/communication/computation cost. In these settings, the agent's interaction with its environment includes the additional component of deciding when to sense the state, in a manner that balances the value associated with optimal (state-specific) actions and the cost of sensing. We formulate this as an expected discounted cost Markov Decision Process (MDP), wherein the agent incurs an additional cost for sensing its next state, but has the option to take actions while remaining `blind' to the system state. We pose this problem as a classical discounted cost MDP with an expanded (countably infinite) state space. While computing the optimal policy for this MDP is intractable in general, we derive lower bounds on the optimal value function, which allow us to bound the suboptimality gap of any policy. We also propose a computationally efficient algorithm SPI, based on policy improvement, which in practice performs close to the optimal policy. Finally, we benchmark against the state-of-the-art via a numerical case study.

MDPs with a State Sensing Cost

TL;DR

This work addresses MDPs with an explicit cost for state sensing, formulating an augmented, countably infinite MDP \\mathcal{M}_k\\ that allows the agent to sense or remain blind at each step. It introduces SPI, a selective policy-improvement algorithm that greedily searches for improving blind-action sequences and leverages a myopic sensing value to guide updates, achieving near-optimal performance in practice. The authors establish provable guarantees through a threshold on the sensing cost for always-sense optimality and a truncation framework that yields computable lower bounds and suboptimality measures, enabling tractable planning for large or continuous state spaces. Empirically, SPI outperforms state-of-the-art POMDP solvers and the ATM heuristic across diverse domains (e.g., ICU-Sepsis, Frozen Lake, Taxi) with reasonable compute times, and extensions to non-uniform costs and continuous spaces are discussed, highlighting practical impact for healthcare, robotics, and sensor networks.

Abstract

In many practical sequential decision-making problems, tracking the state of the environment incurs a sensing/communication/computation cost. In these settings, the agent's interaction with its environment includes the additional component of deciding when to sense the state, in a manner that balances the value associated with optimal (state-specific) actions and the cost of sensing. We formulate this as an expected discounted cost Markov Decision Process (MDP), wherein the agent incurs an additional cost for sensing its next state, but has the option to take actions while remaining `blind' to the system state. We pose this problem as a classical discounted cost MDP with an expanded (countably infinite) state space. While computing the optimal policy for this MDP is intractable in general, we derive lower bounds on the optimal value function, which allow us to bound the suboptimality gap of any policy. We also propose a computationally efficient algorithm SPI, based on policy improvement, which in practice performs close to the optimal policy. Finally, we benchmark against the state-of-the-art via a numerical case study.
Paper Structure (25 sections, 5 theorems, 63 equations, 5 figures, 3 tables, 2 algorithms)

This paper contains 25 sections, 5 theorems, 63 equations, 5 figures, 3 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Problem Formulation
  3. Selective Policy Improvement
  4. Step-by-Step Walkthrough of $\mathsf{Policy Update}$
  5. Solving $\mathcal{M}_k$ with Provable Guarantees
  6. Optimality of always sensing
  7. Analysis via state space truncation
  8. Numerical Case Studies
  9. Discussion and Conclusion
  10. Proofs
  11. Proof of Theorem \ref{['thm:always_sense']}
  12. Proof of Theorem \ref{['thm:depththreshold']}
  13. Proof of Lemma \ref{['thm:one-step']}
  14. Proof of Theorem \ref{['thm:pi_N_optimal']}
  15. Proof of Lemma \ref{['thm:epsilon']}
  16. ...and 10 more sections

Key Result

Theorem 4.1

If then the AS policy defined in Section sec:heuristic is optimal for the MDP $\mathcal{M}_{k}.$

Figures (5)

  • Figure 1: State space of $\mathcal{M}_{k,2}$ for a 2-state 2-action baseline MDP $\mathcal{M}$
  • Figure 2: A two-state two-action baseline MDP with actions $\{Red, Blue\}$ and $\alpha = 0.5$.
  • Figure 3: Applying Thm \ref{['thm:pi_N_optimal']} on MDP in Fig \ref{['fig:example2']}; $k = 0.25$
  • Figure 4: Two-state two-action MDP with actions $\{Red, Blue\}$, $k =0.005$ and $\alpha = 0.5$
  • Figure 5: Applying Thm \ref{['thm:pi_N_optimal']} to Inventory Management

Theorems & Definitions (9)

  • Theorem 4.1
  • Theorem 4.2
  • Lemma 4.3
  • Theorem 4.4
  • proof
  • proof
  • proof
  • proof
  • Lemma D.1