Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Off-policy Evaluation in Doubly Inhomogeneous Environments

Zeyu Bian, Chengchun Shi, Zhengling Qi, Lan Wang

TL;DR

The paper addresses off-policy evaluation in environments with both temporal nonstationarity and individual heterogeneity by introducing a latent-factor, two-way inhomogeneous decision process and developing both model-free and model-based OPE methods. The model-free approach leverages backward induction with a two-way fixed-effects (and optional interactive) structure and linear sieves to estimate time- and subject-specific policy values, while the model-based approach learns a latent-mixture transition model via EM and simulates outcomes under the target policy. Theoretical results provide convergence rates and asymptotic normality for the estimators, and extensive simulations plus a real-data MIMIC-III study demonstrate competitive performance and practical relevance to healthcare decision-making. This work extends OPE to doubly inhomogeneous settings, enabling time- and individual-specific policy evaluation with statistically sound guarantees and applicable methodologies for offline RL in complex, real-world data.

Abstract

This work aims to study off-policy evaluation (OPE) under scenarios where two key reinforcement learning (RL) assumptions -- temporal stationarity and individual homogeneity are both violated. To handle the ``double inhomogeneities", we propose a class of latent factor models for the reward and observation transition functions, under which we develop a general OPE framework that consists of both model-based and model-free approaches. To our knowledge, this is the first paper that develops statistically sound OPE methods in offline RL with double inhomogeneities. It contributes to a deeper understanding of OPE in environments, where standard RL assumptions are not met, and provides several practical approaches in these settings. We establish the theoretical properties of the proposed value estimators and empirically show that our approach outperforms competing methods that ignore either temporal nonstationarity or individual heterogeneity. Finally, we illustrate our method on a data set from the Medical Information Mart for Intensive Care.

Off-policy Evaluation in Doubly Inhomogeneous Environments

TL;DR

The paper addresses off-policy evaluation in environments with both temporal nonstationarity and individual heterogeneity by introducing a latent-factor, two-way inhomogeneous decision process and developing both model-free and model-based OPE methods. The model-free approach leverages backward induction with a two-way fixed-effects (and optional interactive) structure and linear sieves to estimate time- and subject-specific policy values, while the model-based approach learns a latent-mixture transition model via EM and simulates outcomes under the target policy. Theoretical results provide convergence rates and asymptotic normality for the estimators, and extensive simulations plus a real-data MIMIC-III study demonstrate competitive performance and practical relevance to healthcare decision-making. This work extends OPE to doubly inhomogeneous settings, enabling time- and individual-specific policy evaluation with statistically sound guarantees and applicable methodologies for offline RL in complex, real-world data.

Abstract

This work aims to study off-policy evaluation (OPE) under scenarios where two key reinforcement learning (RL) assumptions -- temporal stationarity and individual homogeneity are both violated. To handle the ``double inhomogeneities", we propose a class of latent factor models for the reward and observation transition functions, under which we develop a general OPE framework that consists of both model-based and model-free approaches. To our knowledge, this is the first paper that develops statistically sound OPE methods in offline RL with double inhomogeneities. It contributes to a deeper understanding of OPE in environments, where standard RL assumptions are not met, and provides several practical approaches in these settings. We establish the theoretical properties of the proposed value estimators and empirically show that our approach outperforms competing methods that ignore either temporal nonstationarity or individual heterogeneity. Finally, we illustrate our method on a data set from the Medical Information Mart for Intensive Care.
Paper Structure (9 sections, 3 theorems, 28 equations, 1 figure, 4 tables, 1 algorithm)

This paper contains 9 sections, 3 theorems, 28 equations, 1 figure, 4 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Two-way Doubly Inhomogeneous Decision Processes
  3. Model-free OPE
  4. A Linear Sieve Estimator for Two-way Fixed Effects Model
  5. Beyond the additivity assumption
  6. Model-based OPE
  7. Theoretical Results
  8. Simulation Studies
  9. Real Data Analysis

Key Result

Proposition 1

For any integer $k<t$, the Q-function $Q^{\pi}_{i,t-k+1,t}(o,a)$ satisfies where $\theta^{\pi}_{k,i}$ and $\lambda^{\pi}_{k,t}$ depend only on $i, k, \pi$ and $t,k, \pi$ respectively.

Figures (1)

  • Figure 1: The estimated value function of $\eta^{\pi}_i$ and $\eta^{\pi}_t$ under three target policies, where $\pi_1$ is the always high dose policy, $\pi_2$ is always low dose policy, and $\pi_3$ is the tailored policy.

Theorems & Definitions (8)

  • Remark 1
  • Remark 2
  • Remark 3
  • Remark 4
  • Remark 5
  • Proposition 1
  • Theorem 1: Rates of Convergence
  • Theorem 2: Asymptotically Normality