Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Sample Complexity of Offline Distributionally Robust Linear Markov Decision Processes

He Wang, Laixi Shi, Yuejie Chi

TL;DR

A pessimistic model-based algorithm is developed and its sample complexity bound under minimal data coverage assumptions is established, which outperforms prior art by at least $\widetilde{O}(d)$, where $d$ is the feature dimension.

Abstract

In offline reinforcement learning (RL), the absence of active exploration calls for attention on the model robustness to tackle the sim-to-real gap, where the discrepancy between the simulated and deployed environments can significantly undermine the performance of the learned policy. To endow the learned policy with robustness in a sample-efficient manner in the presence of high-dimensional state-action space, this paper considers the sample complexity of distributionally robust linear Markov decision processes (MDPs) with an uncertainty set characterized by the total variation distance using offline data. We develop a pessimistic model-based algorithm and establish its sample complexity bound under minimal data coverage assumptions, which outperforms prior art by at least $\widetilde{O}(d)$, where $d$ is the feature dimension. We further improve the performance guarantee of the proposed algorithm by incorporating a carefully-designed variance estimator.

Sample Complexity of Offline Distributionally Robust Linear Markov Decision Processes

TL;DR

A pessimistic model-based algorithm is developed and its sample complexity bound under minimal data coverage assumptions is established, which outperforms prior art by at least , where is the feature dimension.

Abstract

In offline reinforcement learning (RL), the absence of active exploration calls for attention on the model robustness to tackle the sim-to-real gap, where the discrepancy between the simulated and deployed environments can significantly undermine the performance of the learned policy. To endow the learned policy with robustness in a sample-efficient manner in the presence of high-dimensional state-action space, this paper considers the sample complexity of distributionally robust linear Markov decision processes (MDPs) with an uncertainty set characterized by the total variation distance using offline data. We develop a pessimistic model-based algorithm and establish its sample complexity bound under minimal data coverage assumptions, which outperforms prior art by at least , where is the feature dimension. We further improve the performance guarantee of the proposed algorithm by incorporating a carefully-designed variance estimator.
Paper Structure (62 sections, 25 theorems, 217 equations, 1 table, 4 algorithms)

This paper contains 62 sections, 25 theorems, 217 equations, 1 table, 4 algorithms.

Table of Contents

  1. Introduction
  2. Main contribution
  3. Related works
  4. Finite-sample guarantees for linear MDPs.
  5. Distributionally robust RL with tabular MDPs.
  6. Distributionally robust RL with linear MDPs.
  7. Notation.
  8. Problem Setup
  9. Standard linear MDPs.
  10. Lin-RMDPs: distributionally robust linear MDPs.
  11. Robust value function and robust Bellman equations.
  12. Offline data.
  13. Learning goal.
  14. Algorithm and Performance Guarantees
  15. Empirical robust Bellman operator and strong duality
  16. ...and 47 more sections

Key Result

Lemma 1

Suppose that the finite-horizon Lin-RMDPs satisfies Assumption assump:linear-mdp and assump:d_rectangular. There exist weights $w^{\rho} = \{w^{\rho}_h\}_{h = 1}^H$, where $w^{\rho}_h \vcentcolon= \theta_h + \inf_{\mu_h\in \mathcal{U}^{\rho}(\mu_h^0)} \int_{ \mathcal{S}} \mu_h(s') f(s') {\mathrm{d}}

Theorems & Definitions (30)

  • Lemma 1: Linearity of robust Bellman operators
  • Theorem 1
  • Corollary 1: Partial feature coverage
  • Corollary 2: Full feature coverage
  • Theorem 2
  • Lemma 2: Hoeffding-type inequality for self-normalized process abbasi2011improved
  • Lemma 3: Bernstein-type inequality for self-normalized process zhou21variance
  • Lemma 4: Lemma H.5, min2021variance
  • Lemma 5: Lemma 5.1, jin2021pessimism
  • Lemma 6
  • ...and 20 more