Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Demystifying the Paradox of Importance Sampling with an Estimated History-Dependent Behavior Policy in Off-Policy Evaluation

Hongyi Zhou, Josiah P. Hanna, Jin Zhu, Ying Yang, Chengchun Shi

TL;DR

This paper tackles off-policy evaluation (OPE) when the behavior policy is estimated from data, addressing the paradox that history-dependent behavior policies can reduce IS-based MSE even when the true policy is Markov. It develops a bias-variance decomposition for ordinary IS (OIS) and extends the analysis to sequential IS (SIS), doubly robust (DR), and marginalized IS (MIS) estimators, showing that longer history can reduce asymptotic variance but increases finite-sample bias. The authors also extend these results to nonparametric (sieve) estimators and demonstrate the findings through numerical experiments in CartPole and other MuJoCo environments, highlighting a consistent large-sample gain from history while warning about finite-sample bias and MIS pitfalls. Finally, the paper offers practical guidelines for choosing history length via a variance-based criterion and discusses implications for OPE practice and future extension to broader settings.

Abstract

This paper studies off-policy evaluation (OPE) in reinforcement learning with a focus on behavior policy estimation for importance sampling. Prior work has shown empirically that estimating a history-dependent behavior policy can lead to lower mean squared error (MSE) even when the true behavior policy is Markovian. However, the question of why the use of history should lower MSE remains open. In this paper, we theoretically demystify this paradox by deriving a bias-variance decomposition of the MSE of ordinary importance sampling (IS) estimators, demonstrating that history-dependent behavior policy estimation decreases their asymptotic variances while increasing their finite-sample biases. Additionally, as the estimated behavior policy conditions on a longer history, we show a consistent decrease in variance. We extend these findings to a range of other OPE estimators, including the sequential IS estimator, the doubly robust estimator and the marginalized IS estimator, with the behavior policy estimated either parametrically or non-parametrically.

Demystifying the Paradox of Importance Sampling with an Estimated History-Dependent Behavior Policy in Off-Policy Evaluation

TL;DR

This paper tackles off-policy evaluation (OPE) when the behavior policy is estimated from data, addressing the paradox that history-dependent behavior policies can reduce IS-based MSE even when the true policy is Markov. It develops a bias-variance decomposition for ordinary IS (OIS) and extends the analysis to sequential IS (SIS), doubly robust (DR), and marginalized IS (MIS) estimators, showing that longer history can reduce asymptotic variance but increases finite-sample bias. The authors also extend these results to nonparametric (sieve) estimators and demonstrate the findings through numerical experiments in CartPole and other MuJoCo environments, highlighting a consistent large-sample gain from history while warning about finite-sample bias and MIS pitfalls. Finally, the paper offers practical guidelines for choosing history length via a variance-based criterion and discusses implications for OPE practice and future extension to broader settings.

Abstract

This paper studies off-policy evaluation (OPE) in reinforcement learning with a focus on behavior policy estimation for importance sampling. Prior work has shown empirically that estimating a history-dependent behavior policy can lead to lower mean squared error (MSE) even when the true behavior policy is Markovian. However, the question of why the use of history should lower MSE remains open. In this paper, we theoretically demystify this paradox by deriving a bias-variance decomposition of the MSE of ordinary importance sampling (IS) estimators, demonstrating that history-dependent behavior policy estimation decreases their asymptotic variances while increasing their finite-sample biases. Additionally, as the estimated behavior policy conditions on a longer history, we show a consistent decrease in variance. We extend these findings to a range of other OPE estimators, including the sequential IS estimator, the doubly robust estimator and the marginalized IS estimator, with the behavior policy estimated either parametrically or non-parametrically.

Paper Structure

This paper contains 19 sections, 9 theorems, 92 equations, 4 figures, 1 table.

Table of Contents

  1. Introduction
  2. Literature review on OPE
  3. Building intuition: from bandits to MDPs
  4. A bandit example
  5. OPE in MDPs
  6. Demystifying the paradox in MDPs
  7. Ordinary IS estimator
  8. Sequential IS estimator
  9. Doubly robust estimator
  10. Marginalized importance sampling estimator
  11. Extensions to cases where the behavior policy is estimated nonparametrically
  12. Numerical studies
  13. Discussion
  14. Details of experiments
  15. Additional experiment results
  16. ...and 4 more sections

Key Result

Lemma 1

$\textrm{MSE}_A(\widehat{v}_{\textrm{IS}}^{\textrm{CD}})\le\textrm{MSE}_A(\widehat{v}_{\textrm{IS}}^{\textrm{CA}}) \le\textrm{MSE}_A(\widehat{v}_{\text{IS}}^{\dagger})$. The first equality hold if and only if the reward function $r$ is independent of the context $S$ whereas the second equality holds

Figures (4)

  • Figure 1: The left panel is log absolute bias of the three IS estimators. The right panel shows log MSE of three different estimators. Results are averaged over $10^4$ trials.
  • Figure 2: Absolute bias (left panel) and log MSE (right panel) of three OPE estimators: SIS (top panel), DR (middle panel), MIS (top panel). The results are averaged over 50 simulations.
  • Figure 3: Bias, log variance and log MSE for OIS estimators across three different environments
  • Figure 4: Bias, log variance and log MSE for OIS,DR and MIS estimators in Swimmer environment

Theorems & Definitions (9)

  • Lemma 1
  • Theorem 2
  • Corollary 3
  • Theorem 4
  • Corollary 5
  • Theorem 6
  • Corollary 7
  • Theorem 8
  • Theorem 9