Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Abstract Reward Processes: Leveraging State Abstraction for Consistent Off-Policy Evaluation

Shreyas Chaudhari, Ameet Deshpande, Bruno Castro da Silva, Philip S. Thomas

TL;DR

The best STAR estimator outperforms baselines in all twelve cases studied, and even the median STAR estimator surpasses the baselines in seven out of the twelve cases, demonstrating that estimators within STAR outperform existing methods.

Abstract

Evaluating policies using off-policy data is crucial for applying reinforcement learning to real-world problems such as healthcare and autonomous driving. Previous methods for off-policy evaluation (OPE) generally suffer from high variance or irreducible bias, leading to unacceptably high prediction errors. In this work, we introduce STAR, a framework for OPE that encompasses a broad range of estimators -- which include existing OPE methods as special cases -- that achieve lower mean squared prediction errors. STAR leverages state abstraction to distill complex, potentially continuous problems into compact, discrete models which we call abstract reward processes (ARPs). Predictions from ARPs estimated from off-policy data are provably consistent (asymptotically correct). Rather than proposing a specific estimator, we present a new framework for OPE and empirically demonstrate that estimators within STAR outperform existing methods. The best STAR estimator outperforms baselines in all twelve cases studied, and even the median STAR estimator surpasses the baselines in seven out of the twelve cases.

Abstract Reward Processes: Leveraging State Abstraction for Consistent Off-Policy Evaluation

TL;DR

The best STAR estimator outperforms baselines in all twelve cases studied, and even the median STAR estimator surpasses the baselines in seven out of the twelve cases, demonstrating that estimators within STAR outperform existing methods.

Abstract

Evaluating policies using off-policy data is crucial for applying reinforcement learning to real-world problems such as healthcare and autonomous driving. Previous methods for off-policy evaluation (OPE) generally suffer from high variance or irreducible bias, leading to unacceptably high prediction errors. In this work, we introduce STAR, a framework for OPE that encompasses a broad range of estimators -- which include existing OPE methods as special cases -- that achieve lower mean squared prediction errors. STAR leverages state abstraction to distill complex, potentially continuous problems into compact, discrete models which we call abstract reward processes (ARPs). Predictions from ARPs estimated from off-policy data are provably consistent (asymptotically correct). Rather than proposing a specific estimator, we present a new framework for OPE and empirically demonstrate that estimators within STAR outperform existing methods. The best STAR estimator outperforms baselines in all twelve cases studied, and even the median STAR estimator surpasses the baselines in seven out of the twelve cases.
Paper Structure (42 sections, 10 theorems, 98 equations, 4 figures, 1 algorithm)

This paper contains 42 sections, 10 theorems, 98 equations, 4 figures, 1 algorithm.

Table of Contents

  1. Introduction
  2. Idea Summary:
  3. Contributions:
  4. Background and Notation
  5. Background:
  6. Notation:
  7. Markov Reward Processes
  8. Abstract Reward Processes
  9. Estimation from Off-Policy Data: Weighted Maximum Likelihood Estimation
  10. Off-Policy Evaluation with ARPs
  11. Variance Reduction: Leveraging Markovness of the State Abstraction
  12. Weight Clipping without Asymptotic Bias:
  13. Fantastic $\phi$'s and Where to Find Them
  14. Recovering Existing OPE Methods from STAR:
  15. Empirical Analysis
  16. ...and 27 more sections

Key Result

Theorem 3.1

$\forall ~\phi \in \Phi$, the performance of a policy $\pi$ is equal to the expected return of the abstract reward process $\mathfrak{R}_\phi^\pi$ defined from MDP $M$, i.e., $J(\pi; \mathfrak{R}_\phi^\pi) = J(\pi; M)$.

Figures (4)

  • Figure 1: (a): MDP $M$ and policy $\pi_b$ are transformed into a discrete abstract reward process (ARP) using a state abstraction function $\phi$. The ARP aggregates rewards (denoted by stars) and transition probabilities from all states that map to each abstract state. (b): A model of the ARP for the evaluation policy $\pi_e$ is constructed by: reweighting data generated by $\pi_b$ with importance weights $\rho$ (middle), applying the state abstraction function $\phi$, and performing weighted maximum likelihood estimation of the ARP (right). The expected return of a model of this ARP estimated from off-policy data is a consistent estimator of the expected return of $\pi_e$.
  • Figure 2: Mean squared prediction errors of the estimated ARPs for the set of hyperparameters swept over for CartPole.
  • Figure 3: Mean squared prediction errors of best and median ARPs from STAR compared against existing OPE methods. The empirically estimated bias-variance decomposition of the error is shown. The results are averaged over 200 trials, with error bars indicating standard error. Note: For ICU-Sepsis, regression-based methods (MRDR and Q-Reg) were computationally intractable due to the large state set, as the corresponding Weighted Least Squares methods for regression were too slow. In all domains and across all datasizes, the best ARP in STAR outperforms baselines in all cases, and the even the median estimator does so in 7 out of 12 cases.
  • Figure 4: Heatmap of the log mean squared error (MSE) of the OPE estimators for the CartPole, Asterix, and ICU-Sepsis domains. The vertical axis represents the number of abstract states $|\mathcal{Z}|$, and the horizontal axis represents the value of the hyperparameter $c$. The color intensity indicates the log MSE, with lower values denoting better performance. Note that the variation with $|\mathcal{Z}|$ is strongly influenced by the class of abstraction functions used, which in this work is CluSTAR.

Theorems & Definitions (27)

  • Theorem 3.1
  • proof
  • Lemma 3.1
  • proof
  • Lemma 3.1
  • proof
  • Theorem 4.1
  • proof
  • Definition 4.2: $c$-th order Markov
  • Theorem 4.3
  • ...and 17 more