Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Learning Infinite-Horizon Average-Reward Linear Mixture MDPs of Bounded Span

Woojin Chae, Kihyuk Hong, Yufan Zhang, Ambuj Tewari, Dabeen Lee

TL;DR

This paper applies the recently developed technique of running value iteration on a discounted-reward MDP approximation with clipping by the span to achieve a nearly minimax optimal regret upper bound, and proves that the value iteration procedure, even with the clipping operation, converges.

Abstract

This paper proposes a computationally tractable algorithm for learning infinite-horizon average-reward linear mixture Markov decision processes (MDPs) under the Bellman optimality condition. Our algorithm for linear mixture MDPs achieves a nearly minimax optimal regret upper bound of $\widetilde{\mathcal{O}}(d\sqrt{\mathrm{sp}(v^*)T})$ over $T$ time steps where $\mathrm{sp}(v^*)$ is the span of the optimal bias function $v^*$ and $d$ is the dimension of the feature mapping. Our algorithm applies the recently developed technique of running value iteration on a discounted-reward MDP approximation with clipping by the span. We prove that the value iteration procedure, even with the clipping operation, converges. Moreover, we show that the associated variance term due to random transitions can be bounded even under clipping. Combined with the weighted ridge regression-based parameter estimation scheme, this leads to the nearly minimax optimal regret guarantee.

Learning Infinite-Horizon Average-Reward Linear Mixture MDPs of Bounded Span

TL;DR

This paper applies the recently developed technique of running value iteration on a discounted-reward MDP approximation with clipping by the span to achieve a nearly minimax optimal regret upper bound, and proves that the value iteration procedure, even with the clipping operation, converges.

Abstract

This paper proposes a computationally tractable algorithm for learning infinite-horizon average-reward linear mixture Markov decision processes (MDPs) under the Bellman optimality condition. Our algorithm for linear mixture MDPs achieves a nearly minimax optimal regret upper bound of over time steps where is the span of the optimal bias function and is the dimension of the feature mapping. Our algorithm applies the recently developed technique of running value iteration on a discounted-reward MDP approximation with clipping by the span. We prove that the value iteration procedure, even with the clipping operation, converges. Moreover, we show that the associated variance term due to random transitions can be bounded even under clipping. Combined with the weighted ridge regression-based parameter estimation scheme, this leads to the nearly minimax optimal regret guarantee.

Paper Structure

This paper contains 35 sections, 17 theorems, 126 equations, 2 figures, 1 table.

Table of Contents

  1. Introduction
  2. Related Work
  3. Reinforcement Learning with Linear Function Approximation
  4. Infinite-Horizon Average-Reward Reinforcement Learning
  5. Preliminaries
  6. Notations
  7. Infinite-Horizon Average-Reward MDP
  8. Discounted-Reward MDP
  9. Linear Mixture MDPs
  10. The Proposed Algorithm
  11. Discounted Value Iteration
  12. Clipping Operation
  13. Variance-Aware Ridge Regression-Based Parameter Estimation
  14. Regret Analysis of UCLK-C
  15. Regret Term $I_1$
  16. ...and 20 more sections

Key Result

Lemma 4.1

zhou-mixture-finite-optimal Let $\{\mathcal{G}_t\}_{t=1}^{\infty}$ be a filtration, $\{{x}_t, \eta_t\}_{t\geq1}$ a stochastic process such that ${x}_t\in\mathbb{R}^d$ is $\mathcal{G}_t$-measurable while $\eta_t\in\mathbb{R}$ is $\mathcal{G}_{t+1}$-measurable. For $t \geq 1$, let $y_t = \langle {x}_t where $\beta_t=8\sigma\sqrt{d\log(1+tL^2/(d\lambda))\log(4t^2/\delta)}+4R\log(4t^2/\delta)$, $\mu_t

Figures (2)

  • Figure 1: Illustration of the Hard-to-Learn Infinite-Horizon MDP Instance
  • Figure 2: Regret comparison of UCLK-C and UCRL2-VTR (Bernstein-type), $\delta=1/120$$(\Leftrightarrow D=120)$

Theorems & Definitions (18)

  • Lemma 4.1
  • Lemma 4.2
  • Theorem 1
  • Lemma 5.1
  • Lemma 5.2
  • Lemma 5.3
  • Lemma 5.4
  • Lemma 5.5
  • Lemma 5.6
  • Lemma 5.7
  • ...and 8 more