Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Robust Offline Reinforcement Learning for Non-Markovian Decision Processes

Ruiquan Huang, Yingbin Liang, Jing Yang

TL;DR

This paper proposes a new algorithm, featuring a novel dataset distillation and a lower confidence bound (LCB) design for robust values under different types of the uncertainty set, and extends it to the case when the nominal model does not have specific structure.

Abstract

Distributionally robust offline reinforcement learning (RL) aims to find a policy that performs the best under the worst environment within an uncertainty set using an offline dataset collected from a nominal model. While recent advances in robust RL focus on Markov decision processes (MDPs), robust non-Markovian RL is limited to planning problem where the transitions in the uncertainty set are known. In this paper, we study the learning problem of robust offline non-Markovian RL. Specifically, when the nominal model admits a low-rank structure, we propose a new algorithm, featuring a novel dataset distillation and a lower confidence bound (LCB) design for robust values under different types of the uncertainty set. We also derive new dual forms for these robust values in non-Markovian RL, making our algorithm more amenable to practical implementation. By further introducing a novel type-I concentrability coefficient tailored for offline low-rank non-Markovian decision processes, we prove that our algorithm can find an $ε$-optimal robust policy using $O(1/ε^2)$ offline samples. Moreover, we extend our algorithm to the case when the nominal model does not have specific structure. With a new type-II concentrability coefficient, the extended algorithm also enjoys polynomial sample efficiency under all different types of the uncertainty set.

Robust Offline Reinforcement Learning for Non-Markovian Decision Processes

TL;DR

This paper proposes a new algorithm, featuring a novel dataset distillation and a lower confidence bound (LCB) design for robust values under different types of the uncertainty set, and extends it to the case when the nominal model does not have specific structure.

Abstract

Distributionally robust offline reinforcement learning (RL) aims to find a policy that performs the best under the worst environment within an uncertainty set using an offline dataset collected from a nominal model. While recent advances in robust RL focus on Markov decision processes (MDPs), robust non-Markovian RL is limited to planning problem where the transitions in the uncertainty set are known. In this paper, we study the learning problem of robust offline non-Markovian RL. Specifically, when the nominal model admits a low-rank structure, we propose a new algorithm, featuring a novel dataset distillation and a lower confidence bound (LCB) design for robust values under different types of the uncertainty set. We also derive new dual forms for these robust values in non-Markovian RL, making our algorithm more amenable to practical implementation. By further introducing a novel type-I concentrability coefficient tailored for offline low-rank non-Markovian decision processes, we prove that our algorithm can find an -optimal robust policy using offline samples. Moreover, we extend our algorithm to the case when the nominal model does not have specific structure. With a new type-II concentrability coefficient, the extended algorithm also enjoys polynomial sample efficiency under all different types of the uncertainty set.

Paper Structure

This paper contains 21 sections, 22 theorems, 120 equations, 2 algorithms.

Table of Contents

  1. Introduction
  2. Related Work
  3. Preliminaries
  4. Non-Markovian Decision Processes
  5. Robust RL
  6. Low-rank Non-Markovian Decision Process
  7. Robust Offline RL with Low-rank Nominal Model
  8. Algorithm Design
  9. Theoretical Guarantee
  10. Robust Offline RL under General Non-Markovian Decision Processes
  11. Algorithm Design
  12. Theoretical Guarantee
  13. Conclusion
  14. Technical Lemmas
  15. Dual Form of Robust Values
  16. ...and 6 more sections

Key Result

Lemma 4.1

For any $\theta$, let $V_{B(\theta), R,H+1}^{\pi}(\tau_H) =0$. Then, we have

Theorems & Definitions (43)

  • Definition 3.1: Rank-$r$ non-Markov decision process
  • Lemma 4.1
  • Definition 4.2: $\varepsilon$-Bracketing number of $\Theta$
  • Remark 4.3
  • Remark 4.6
  • Theorem 4.7
  • Remark 5.1
  • Theorem 5.2
  • Proposition A.1
  • Proposition A.2
  • ...and 33 more