Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Achieving Constant Regret in Linear Markov Decision Processes

Weitong Zhang, Zhiyuan Fan, Jiafan He, Quanquan Gu

TL;DR

To the best of the knowledge, Cert-LSVI-UCB is the first algorithm to achieve a constant, instance-dependent, high-probability regret bound in RL with linear function approximation without relying on prior distribution assumptions.

Abstract

We study the constant regret guarantees in reinforcement learning (RL). Our objective is to design an algorithm that incurs only finite regret over infinite episodes with high probability. We introduce an algorithm, Cert-LSVI-UCB, for misspecified linear Markov decision processes (MDPs) where both the transition kernel and the reward function can be approximated by some linear function up to misspecification level $ζ$. At the core of Cert-LSVI-UCB is an innovative \method, which facilitates a fine-grained concentration analysis for multi-phase value-targeted regression, enabling us to establish an instance-dependent regret bound that is constant w.r.t. the number of episodes. Specifically, we demonstrate that for a linear MDP characterized by a minimal suboptimality gap $Δ$, Cert-LSVI-UCB has a cumulative regret of $\tilde{\mathcal{O}}(d^3H^5/Δ)$ with high probability, provided that the misspecification level $ζ$ is below $\tilde{\mathcal{O}}(Δ/ (\sqrt{d}H^2))$. Here $d$ is the dimension of the feature space and $H$ is the horizon. Remarkably, this regret bound is independent of the number of episodes $K$. To the best of our knowledge, Cert-LSVI-UCB is the first algorithm to achieve a constant, instance-dependent, high-probability regret bound in RL with linear function approximation without relying on prior distribution assumptions.

Achieving Constant Regret in Linear Markov Decision Processes

TL;DR

To the best of the knowledge, Cert-LSVI-UCB is the first algorithm to achieve a constant, instance-dependent, high-probability regret bound in RL with linear function approximation without relying on prior distribution assumptions.

Abstract

We study the constant regret guarantees in reinforcement learning (RL). Our objective is to design an algorithm that incurs only finite regret over infinite episodes with high probability. We introduce an algorithm, Cert-LSVI-UCB, for misspecified linear Markov decision processes (MDPs) where both the transition kernel and the reward function can be approximated by some linear function up to misspecification level . At the core of Cert-LSVI-UCB is an innovative \method, which facilitates a fine-grained concentration analysis for multi-phase value-targeted regression, enabling us to establish an instance-dependent regret bound that is constant w.r.t. the number of episodes. Specifically, we demonstrate that for a linear MDP characterized by a minimal suboptimality gap , Cert-LSVI-UCB has a cumulative regret of with high probability, provided that the misspecification level is below . Here is the dimension of the feature space and is the horizon. Remarkably, this regret bound is independent of the number of episodes . To the best of our knowledge, Cert-LSVI-UCB is the first algorithm to achieve a constant, instance-dependent, high-probability regret bound in RL with linear function approximation without relying on prior distribution assumptions.
Paper Structure (60 sections, 48 theorems, 167 equations, 2 figures, 3 tables, 2 algorithms)

This paper contains 60 sections, 48 theorems, 167 equations, 2 figures, 3 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Related Work
  3. Instance-dependent regret bound in RL.
  4. RL with model misspecification.
  5. Preliminaries
  6. Proposed Algorithms
  7. Main algorithm: Cert-LSVI-UCB
  8. Subroutine: Cert-LinUCB
  9. Constant Regret Guarantee
  10. Technical Challenges and Highlight of Proof Techniques
  11. Challenge 1. Achieving layer-wise local estimation error.
  12. Our approach: Cert-LinUCB.
  13. Challenge 2. Achieving constant regret from local estimation error
  14. Our approach: A fine-grained concentration analysis
  15. Conclusions and Limitations
  16. ...and 45 more sections

Key Result

Proposition 3.2

For a $\zeta$-approximate linear MDP, for any policy $\pi$, there exist corresponding weights $\{\mathbf{w}_h^\pi\}_{h \in [H]}$ where $\mathbf{w}_h^\pi = \bm{\theta}_h + \int V_{h+1}^\pi(s') \mathrm d\bm{\mu}_h(s')$ such that for any $(s, a, h) \in {\mathcal{S}} \times \mathcal{A} \times [H]$, $|Q_

Figures (2)

  • Figure 1: Cumulative regret over 2000 episodes with respect to different misspecification level $\zeta$. The result is averaged over 16 individual environments.
  • Figure 2: Cumulative regret with respect to the number of episodes. We reported the median cumulative regret with the shadow area as the region from 25% percentage to 75% percentage statistics over 16 runs.

Theorems & Definitions (57)

  • Proposition 3.2: Lemma C.1, jin2020provably
  • Definition 3.3: Minimal suboptimality gap
  • Theorem 5.1
  • Remark 5.2
  • Remark 5.3
  • Lemma 6.1: Lemma \ref{['lm:concentration-V']}, informal
  • Lemma 6.2: Lemma \ref{['lm:final-value-func-pre']}, informal
  • Lemma 6.3: Lemma \ref{['lm:interplay']}, informal
  • Lemma 6.4: Lemma \ref{['lm:main']}, Informal
  • Remark 6.5
  • ...and 47 more