Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Reinforcement Learning with Continuous Actions Under Unmeasured Confounding

Yuhan Li, Eugene Han, Yifan Hu, Wenzhuo Zhou, Zhengling Qi, Yifan Cui, Ruoqing Zhu

TL;DR

This work tackles offline policy learning in continuous-action settings when unmeasured confounders are present by extending proximal causal inference to infinite-horizon confounded POMDPs. It provides a nonparametric identification framework using $Q$-bridge and $V$-bridge functions, enabling unbiased policy-value estimation $J(\pi)=\mathbb{E}[V^{\pi}(O_0,W_0)]$ from batch data. A minimax estimator for the bridge functions is developed, along with an FQE-like algorithm and a policy-gradient method to optimize within a chosen policy class, with proven consistency, finite-sample guarantees, and regret bounds. The approach is validated through simulations and a real-world Pairfam dataset analysis, showing reduced bias from unmeasured confounding and interpretable, continuous-action policies. Overall, the paper advances offline RL under confounding with continuous actions, offering principled identification, robust estimation, and scalable policy optimization techniques.

Abstract

This paper addresses the challenge of offline policy learning in reinforcement learning with continuous action spaces when unmeasured confounders are present. While most existing research focuses on policy evaluation within partially observable Markov decision processes (POMDPs) and assumes discrete action spaces, we advance this field by establishing a novel identification result to enable the nonparametric estimation of policy value for a given target policy under an infinite-horizon framework. Leveraging this identification, we develop a minimax estimator and introduce a policy-gradient-based algorithm to identify the in-class optimal policy that maximizes the estimated policy value. Furthermore, we provide theoretical results regarding the consistency, finite-sample error bound, and regret bound of the resulting optimal policy. Extensive simulations and a real-world application using the German Family Panel data demonstrate the effectiveness of our proposed methodology.

Reinforcement Learning with Continuous Actions Under Unmeasured Confounding

TL;DR

This work tackles offline policy learning in continuous-action settings when unmeasured confounders are present by extending proximal causal inference to infinite-horizon confounded POMDPs. It provides a nonparametric identification framework using -bridge and -bridge functions, enabling unbiased policy-value estimation from batch data. A minimax estimator for the bridge functions is developed, along with an FQE-like algorithm and a policy-gradient method to optimize within a chosen policy class, with proven consistency, finite-sample guarantees, and regret bounds. The approach is validated through simulations and a real-world Pairfam dataset analysis, showing reduced bias from unmeasured confounding and interpretable, continuous-action policies. Overall, the paper advances offline RL under confounding with continuous actions, offering principled identification, robust estimation, and scalable policy optimization techniques.

Abstract

This paper addresses the challenge of offline policy learning in reinforcement learning with continuous action spaces when unmeasured confounders are present. While most existing research focuses on policy evaluation within partially observable Markov decision processes (POMDPs) and assumes discrete action spaces, we advance this field by establishing a novel identification result to enable the nonparametric estimation of policy value for a given target policy under an infinite-horizon framework. Leveraging this identification, we develop a minimax estimator and introduce a policy-gradient-based algorithm to identify the in-class optimal policy that maximizes the estimated policy value. Furthermore, we provide theoretical results regarding the consistency, finite-sample error bound, and regret bound of the resulting optimal policy. Extensive simulations and a real-world application using the German Family Panel data demonstrate the effectiveness of our proposed methodology.
Paper Structure (10 sections, 8 theorems, 27 equations, 4 figures, 3 tables, 2 algorithms)

This paper contains 10 sections, 8 theorems, 27 equations, 4 figures, 3 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Preliminaries
  3. Methodology
  4. Identification Results
  5. Minimax Estimation
  6. Policy Learning
  7. Theoretical Results
  8. Simulation Studies
  9. Application to Pairfam Dataset
  10. Discussion

Key Result

Theorem 3.1

(Identification) For a confounded POMDP model whose variables satisfy Assumptions markov-complete and some regularity conditions, there always exist $Q$-bridges and $V$-bridges satisfying qbridge and vbridge respectively. Additionally, one particular $Q$-bridge and $V$-bridge can be obtained by solv

Figures (4)

  • Figure 1: DAG of the proposed confounded POMDP (left) and classical POMDP (right).
  • Figure 2: Logarithms of relative MSEs of the proposed (blue sqaures), MDPW (orange circles), and MDP (green triangles) estimators and their associated 95% confidence interval, based on 50 simulations, with different choices of the sample size and length of trajectory $(n, T)$.
  • Figure 3: Boxplots of discounted reward improvements under 100 simulation runs with $\gamma=0.9$. The first two boxes represent the proposed method using neural network and linear policy class. The remaining boxes show the MDP and MDPW version of competing methods.
  • Figure 4: The estimated optimal policy distribution under typical states. The corresponding states are defined in Table \ref{['scenarios']}.

Theorems & Definitions (8)

  • Theorem 3.1
  • Theorem 3.2
  • Theorem 3.3
  • Theorem 3.4
  • Theorem 4.1
  • Theorem 4.2
  • Proposition 4.1
  • Theorem 4.3