Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

SaVeR: Optimal Data Collection Strategy for Safe Policy Evaluation in Tabular MDP

Subhojyoti Mukherjee, Josiah P. Hanna, Robert Nowak

TL;DR

This work studies safe data collection for policy evaluation in finite-horizon tabular MDPs, aiming to estimate the target value $V^{\pi}(s_1)$ with minimal MSE under a safety constraint that ties cumulative cost to a baseline policy via $(1-\alpha)$. It develops SaVeR, an agnostic algorithm that preserves safety while adaptively allocating samples to reduce variance, and provides finite-sample MSE and regret guarantees of $\widetilde{O}(n^{-3/2})$ under a tractability condition. The paper also proves lower bounds for both constrained and unconstrained settings, showing that unsafe instances can be intractable and identifying when safe data collection is feasible. It extends to DAGs, presents a practical DAG approximation, and validates the approach with comprehensive experiments, highlighting its practical impact for safe, data-efficient policy evaluation in real-world decision systems.

Abstract

In this paper, we study safe data collection for the purpose of policy evaluation in tabular Markov decision processes (MDPs). In policy evaluation, we are given a \textit{target} policy and asked to estimate the expected cumulative reward it will obtain. Policy evaluation requires data and we are interested in the question of what \textit{behavior} policy should collect the data for the most accurate evaluation of the target policy. While prior work has considered behavior policy selection, in this paper, we additionally consider a safety constraint on the behavior policy. Namely, we assume there exists a known default policy that incurs a particular expected cost when run and we enforce that the cumulative cost of all behavior policies ran is better than a constant factor of the cost that would be incurred had we always run the default policy. We first show that there exists a class of intractable MDPs where no safe oracle algorithm with knowledge about problem parameters can efficiently collect data and satisfy the safety constraints. We then define the tractability condition for an MDP such that a safe oracle algorithm can efficiently collect data and using that we prove the first lower bound for this setting. We then introduce an algorithm SaVeR for this problem that approximates the safe oracle algorithm and bound the finite-sample mean squared error of the algorithm while ensuring it satisfies the safety constraint. Finally, we show in simulations that SaVeR produces low MSE policy evaluation while satisfying the safety constraint.

SaVeR: Optimal Data Collection Strategy for Safe Policy Evaluation in Tabular MDP

TL;DR

This work studies safe data collection for policy evaluation in finite-horizon tabular MDPs, aiming to estimate the target value with minimal MSE under a safety constraint that ties cumulative cost to a baseline policy via . It develops SaVeR, an agnostic algorithm that preserves safety while adaptively allocating samples to reduce variance, and provides finite-sample MSE and regret guarantees of under a tractability condition. The paper also proves lower bounds for both constrained and unconstrained settings, showing that unsafe instances can be intractable and identifying when safe data collection is feasible. It extends to DAGs, presents a practical DAG approximation, and validates the approach with comprehensive experiments, highlighting its practical impact for safe, data-efficient policy evaluation in real-world decision systems.

Abstract

In this paper, we study safe data collection for the purpose of policy evaluation in tabular Markov decision processes (MDPs). In policy evaluation, we are given a \textit{target} policy and asked to estimate the expected cumulative reward it will obtain. Policy evaluation requires data and we are interested in the question of what \textit{behavior} policy should collect the data for the most accurate evaluation of the target policy. While prior work has considered behavior policy selection, in this paper, we additionally consider a safety constraint on the behavior policy. Namely, we assume there exists a known default policy that incurs a particular expected cost when run and we enforce that the cumulative cost of all behavior policies ran is better than a constant factor of the cost that would be incurred had we always run the default policy. We first show that there exists a class of intractable MDPs where no safe oracle algorithm with knowledge about problem parameters can efficiently collect data and satisfy the safety constraints. We then define the tractability condition for an MDP such that a safe oracle algorithm can efficiently collect data and using that we prove the first lower bound for this setting. We then introduce an algorithm SaVeR for this problem that approximates the safe oracle algorithm and bound the finite-sample mean squared error of the algorithm while ensuring it satisfies the safety constraint. Finally, we show in simulations that SaVeR produces low MSE policy evaluation while satisfying the safety constraint.
Paper Structure (21 sections, 26 theorems, 169 equations, 3 figures, 1 table, 2 algorithms)

This paper contains 21 sections, 26 theorems, 169 equations, 3 figures, 1 table, 2 algorithms.

Table of Contents

  1. Introduction
  2. Preliminaries
  3. Intractability and Lower Bounds
  4. Unconstrained Oracle
  5. Safe Oracle Algorithm for Safe Data Collection
  6. An Intractable MDP
  7. Agnostic Algorithm for Safe Policy Evaluation
  8. Extension to DAG
  9. Experiments
  10. Conclusions
  11. Appendix
  12. Related Works
  13. Previous results and Probability Tools
  14. Intractable MDP
  15. Tractable MDP and Lower Bounds
  16. ...and 6 more sections

Key Result

Proposition 1

Fix an arbitrary $n > 0$. Then there exists an environment where no algorithm (including the safe oracle $\mathbf{b}^k_*$) can be run that will result in a regret $\mathcal{R}_n = \mathcal{L}_n(\pi, \mathbf{b}^*_k) - \mathcal{L}^*_n(\pi, \mathbf{b}_*)$ of $\widetilde{O}(n^{-3/2})$ while satisfying t

Figures (3)

  • Figure 1: MSE in different settings. The vertical axis (log-scaled) gives MSE and the horizontal axis is the number of episodes (or rounds for bandits). Confidence bars show one standard error.
  • Figure 2: The vertical axis gives cumulative constraint violation and the horizontal axis is the number of episodes/rounds. The $0$-axis is shown in pink. A safe algorithm has its plot below the $0$-axis with the plot showing the cumulative unsafe budget.
  • Figure 4: Tractable Tree MDPs $\mathcal{T}$ and $\mathcal{T}'$. The difference between the two Tree MDPs is highlighted in the square box.

Theorems & Definitions (43)

  • Definition 3.1
  • Proposition 1
  • Theorem 1
  • Remark 4.1
  • Theorem 2
  • Corollary 1
  • Definition 5.1
  • Lemma 5.2
  • Proposition 1
  • Lemma 1.1
  • ...and 33 more