Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Reinforcement Learning for Individual Optimal Policy from Heterogeneous Data

Rui Miao, Babak Shahbaba, Annie Qu

TL;DR

An individualized offline policy optimization framework for heterogeneous time-stationary Markov decision processes (MDPs) with individual latent variables is proposed, and the Penalized Pessimistic Personalized Policy Learning (P4L) algorithm guarantees a fast rate on the average regret under a weak partial coverage assumption on behavior policies.

Abstract

Offline reinforcement learning (RL) aims to find optimal policies in dynamic environments in order to maximize the expected total rewards by leveraging pre-collected data. Learning from heterogeneous data is one of the fundamental challenges in offline RL. Traditional methods focus on learning an optimal policy for all individuals with pre-collected data from a single episode or homogeneous batch episodes, and thus, may result in a suboptimal policy for a heterogeneous population. In this paper, we propose an individualized offline policy optimization framework for heterogeneous time-stationary Markov decision processes (MDPs). The proposed heterogeneous model with individual latent variables enables us to efficiently estimate the individual Q-functions, and our Penalized Pessimistic Personalized Policy Learning (P4L) algorithm guarantees a fast rate on the average regret under a weak partial coverage assumption on behavior policies. In addition, our simulation studies and a real data application demonstrate the superior numerical performance of the proposed method compared with existing methods.

Reinforcement Learning for Individual Optimal Policy from Heterogeneous Data

TL;DR

An individualized offline policy optimization framework for heterogeneous time-stationary Markov decision processes (MDPs) with individual latent variables is proposed, and the Penalized Pessimistic Personalized Policy Learning (P4L) algorithm guarantees a fast rate on the average regret under a weak partial coverage assumption on behavior policies.

Abstract

Offline reinforcement learning (RL) aims to find optimal policies in dynamic environments in order to maximize the expected total rewards by leveraging pre-collected data. Learning from heterogeneous data is one of the fundamental challenges in offline RL. Traditional methods focus on learning an optimal policy for all individuals with pre-collected data from a single episode or homogeneous batch episodes, and thus, may result in a suboptimal policy for a heterogeneous population. In this paper, we propose an individualized offline policy optimization framework for heterogeneous time-stationary Markov decision processes (MDPs). The proposed heterogeneous model with individual latent variables enables us to efficiently estimate the individual Q-functions, and our Penalized Pessimistic Personalized Policy Learning (P4L) algorithm guarantees a fast rate on the average regret under a weak partial coverage assumption on behavior policies. In addition, our simulation studies and a real data application demonstrate the superior numerical performance of the proposed method compared with existing methods.
Paper Structure (28 sections, 9 theorems, 87 equations, 6 figures, 4 tables, 1 algorithm)

This paper contains 28 sections, 9 theorems, 87 equations, 6 figures, 4 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Preliminaries
  3. Notations
  4. Frameworks
  5. Methods
  6. Heterogeneous Latent Variable Model
  7. Pessimistic Policy Learning with Latent Variables
  8. Theory
  9. Assumptions
  10. Oracle Regret Bounds
  11. Feasibility and Regret Bounds for the Dual Problem
  12. Computation
  13. Simulation
  14. A simple environment
  15. Simulation Results
  16. ...and 13 more sections

Key Result

Lemma 1

Under Assumption ass: standard, we have that

Figures (6)

  • Figure 1: Boxplots for values of estimated policies for $n=30,60$ and $T=50,100$ and 200. Red dashed lines separate three groups (a), (b) and (c). In each group, the blue dotted line separates the P4L method of different $K$ from the benchmark methods.
  • Figure 2: OpenAI CartPole environment.
  • Figure 3: Values of learned optimal policies estimated by PerSim.
  • Figure 4: The structure of the neural network in the empirical study.
  • Figure 5: Boxplots for values of estimated policies for $n=30,60$ and $150$ and $T=50,100$ and 200. Red dashed lines separate three groups (a), (b) and (c). In each group, the blue dotted line separates the P4L method of different $K$ from the benchmark methods.
  • ...and 1 more figures

Theorems & Definitions (20)

  • Lemma 1
  • Example 1: Linear MDP bradtke1996linearmelo2007q
  • Definition 1: Covering Number
  • Remark 1
  • Remark 2
  • Theorem 1
  • Theorem 2
  • Theorem 3
  • Corollary 1
  • Theorem 4
  • ...and 10 more