Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Efficient and Sharp Off-Policy Evaluation in Robust Markov Decision Processes

Andrew Bennett, Nathan Kallus, Miruna Oprescu, Wen Sun, Kaiwen Wang

TL;DR

An estimator is developed with several important guarantees: it is semiparametrically efficient, and remains so even when certain necessary nuisance functions, such as worst-case Q-functions, are estimated at slow, nonparametric rates, enabling straightforward statistical inference using Wald confidence intervals.

Abstract

We study the evaluation of a policy under best- and worst-case perturbations to a Markov decision process (MDP), using transition observations from the original MDP, whether they are generated under the same or a different policy. This is an important problem when there is the possibility of a shift between historical and future environments, $\textit{e.g.}$ due to unmeasured confounding, distributional shift, or an adversarial environment. We propose a perturbation model that allows changes in the transition kernel densities up to a given multiplicative factor or its reciprocal, extending the classic marginal sensitivity model (MSM) for single time-step decision-making to infinite-horizon RL. We characterize the sharp bounds on policy value under this model $\unicode{x2013}$ $\textit{i.e.}$, the tightest possible bounds based on transition observations from the original MDP $\unicode{x2013}$ and we study the estimation of these bounds from such transition observations. We develop an estimator with several important guarantees: it is semiparametrically efficient, and remains so even when certain necessary nuisance functions, such as worst-case Q-functions, are estimated at slow, nonparametric rates. Our estimator is also asymptotically normal, enabling straightforward statistical inference using Wald confidence intervals. Moreover, when certain nuisances are estimated inconsistently, the estimator still provides valid, albeit possibly not sharp, bounds on the policy value. We validate these properties in numerical simulations. The combination of accounting for environment shifts from train to test (robustness), being insensitive to nuisance-function estimation (orthogonality), and addressing the challenge of learning from finite samples (inference) together leads to credible and reliable policy evaluation.

Efficient and Sharp Off-Policy Evaluation in Robust Markov Decision Processes

TL;DR

An estimator is developed with several important guarantees: it is semiparametrically efficient, and remains so even when certain necessary nuisance functions, such as worst-case Q-functions, are estimated at slow, nonparametric rates, enabling straightforward statistical inference using Wald confidence intervals.

Abstract

We study the evaluation of a policy under best- and worst-case perturbations to a Markov decision process (MDP), using transition observations from the original MDP, whether they are generated under the same or a different policy. This is an important problem when there is the possibility of a shift between historical and future environments, due to unmeasured confounding, distributional shift, or an adversarial environment. We propose a perturbation model that allows changes in the transition kernel densities up to a given multiplicative factor or its reciprocal, extending the classic marginal sensitivity model (MSM) for single time-step decision-making to infinite-horizon RL. We characterize the sharp bounds on policy value under this model , the tightest possible bounds based on transition observations from the original MDP and we study the estimation of these bounds from such transition observations. We develop an estimator with several important guarantees: it is semiparametrically efficient, and remains so even when certain necessary nuisance functions, such as worst-case Q-functions, are estimated at slow, nonparametric rates. Our estimator is also asymptotically normal, enabling straightforward statistical inference using Wald confidence intervals. Moreover, when certain nuisances are estimated inconsistently, the estimator still provides valid, albeit possibly not sharp, bounds on the policy value. We validate these properties in numerical simulations. The combination of accounting for environment shifts from train to test (robustness), being insensitive to nuisance-function estimation (orthogonality), and addressing the challenge of learning from finite samples (inference) together leads to credible and reliable policy evaluation.
Paper Structure (65 sections, 30 theorems, 118 equations, 2 figures, 2 tables, 4 algorithms)

This paper contains 65 sections, 30 theorems, 118 equations, 2 figures, 2 tables, 4 algorithms.

Table of Contents

  1. Introduction
  2. Related Works
  3. Preliminaries
  4. Background: Non-robust OPE
  5. Robust Q-Function Estimation with Fitted-Q Evaluation
  6. Identification of the worst-case Q-function
  7. Estimating the Robust Q-Function with Robust FQE
  8. Robust w-Function Estimation with Minimax Learning
  9. Identification of $U^-$.
  10. Identification of $w^-$.
  11. Estimating w- with Robust Minimax Indirect Learning
  12. Orthogonal and Efficient Estimator for Robust Policy Value
  13. Theoretical Guarantees of the Orthogonal Estimator
  14. Bringing it all together.
  15. Empirical Evaluation
  16. ...and 50 more sections

Key Result

Lemma 3.0

Set $\tau(s,a)=(*){\Lambda(s,a)+1}^{-1}$. Then, for any $q:\mathcal{S}\times\mathcal{A}\to\mathbb{R}$, where $v(s')=\mathbb{E}_{a'\sim\pi_{\textsf{t}}(s')}[q(s',a')]$, and $\mathbb{E},\operatorname{CVaR}_\tau$ are under the observed kernel $P(\cdot\mid s,a)$.

Figures (2)

  • Figure 1: Lower and upper CVaRs and quantiles ($\beta$) of $v(s')\mid s,a$ distribution.
  • Figure 2: Results of our synthetic data experiments. We show results for our three estimators on all four $\Lambda$ values, over our 10 experiment replications. Above: Box plot summarizing range of policy value estimates for each combination of estimator and $\Lambda$, with Horizontal red dashed lines showing the true worst-case policy values $V^-_{d_1}$. Below: Table summarizing the corresponding MSE of these estimators for the true worst-case policy value, along with one standard deviation errors.

Theorems & Definitions (52)

  • Lemma 3.0
  • Theorem 3.3
  • Lemma 4.0
  • Theorem 4.3
  • Theorem 5.1: (Recentered) Efficient Influence Function
  • Remark 5.2
  • Theorem 5.3: Efficiency of $\widehat{V}_{d_1}^-$
  • Theorem B.1: Efficiency of $\widehat{V}^+_{d_1}$
  • Definition D.1: $\alpha$-smooth functions
  • Lemma D.2: $L^\infty$ Bound for $\alpha$-Smooth Functions
  • ...and 42 more