Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Frozen Policy Iteration: Computationally Efficient RL under Linear $Q^π$ Realizability for Deterministic Dynamics

Yijing Ke, Zihan Zhang, Ruosong Wang

TL;DR

This paper proposes a computationally efficient online RL algorithm, named Frozen Policy Iteration, under the linear $Q^{\pi}$ realizability setting that works for Markov Decision Processes with stochastic initial states, stochastic rewards and deterministic transitions and extends it to the Uniform-PAC setting and to function classes with bounded eluder dimension.

Abstract

We study computationally and statistically efficient reinforcement learning under the linear $Q^π$ realizability assumption, where any policy's $Q$-function is linear in a given state-action feature representation. Prior methods in this setting are either computationally intractable, or require (local) access to a simulator. In this paper, we propose a computationally efficient online RL algorithm, named Frozen Policy Iteration, under the linear $Q^π$ realizability setting that works for Markov Decision Processes (MDPs) with stochastic initial states, stochastic rewards and deterministic transitions. Our algorithm achieves a regret bound of $\widetilde{O}(\sqrt{d^2H^6T})$, where $d$ is the dimensionality of the feature space, $H$ is the horizon length, and $T$ is the total number of episodes. Our regret bound is optimal for linear (contextual) bandits which is a special case of our setting with $H = 1$. Existing policy iteration algorithms under the same setting heavily rely on repeatedly sampling the same state by access to the simulator, which is not implementable in the online setting with stochastic initial states studied in this paper. In contrast, our new algorithm circumvents this limitation by strategically using only high-confidence part of the trajectory data and freezing the policy for well-explored states, which ensures that all data used by our algorithm remains effectively on-policy during the whole course of learning. We further demonstrate the versatility of our approach by extending it to the Uniform-PAC setting and to function classes with bounded eluder dimension.

Frozen Policy Iteration: Computationally Efficient RL under Linear $Q^π$ Realizability for Deterministic Dynamics

TL;DR

This paper proposes a computationally efficient online RL algorithm, named Frozen Policy Iteration, under the linear realizability setting that works for Markov Decision Processes with stochastic initial states, stochastic rewards and deterministic transitions and extends it to the Uniform-PAC setting and to function classes with bounded eluder dimension.

Abstract

We study computationally and statistically efficient reinforcement learning under the linear realizability assumption, where any policy's -function is linear in a given state-action feature representation. Prior methods in this setting are either computationally intractable, or require (local) access to a simulator. In this paper, we propose a computationally efficient online RL algorithm, named Frozen Policy Iteration, under the linear realizability setting that works for Markov Decision Processes (MDPs) with stochastic initial states, stochastic rewards and deterministic transitions. Our algorithm achieves a regret bound of , where is the dimensionality of the feature space, is the horizon length, and is the total number of episodes. Our regret bound is optimal for linear (contextual) bandits which is a special case of our setting with . Existing policy iteration algorithms under the same setting heavily rely on repeatedly sampling the same state by access to the simulator, which is not implementable in the online setting with stochastic initial states studied in this paper. In contrast, our new algorithm circumvents this limitation by strategically using only high-confidence part of the trajectory data and freezing the policy for well-explored states, which ensures that all data used by our algorithm remains effectively on-policy during the whole course of learning. We further demonstrate the versatility of our approach by extending it to the Uniform-PAC setting and to function classes with bounded eluder dimension.
Paper Structure (34 sections, 32 theorems, 68 equations, 2 figures, 1 table, 2 algorithms)

This paper contains 34 sections, 32 theorems, 68 equations, 2 figures, 1 table, 2 algorithms.

Table of Contents

  1. Introduction
  2. Related Work
  3. Linear $Q^{\pi}$ Realizability.
  4. Linear $Q^*$ Realizability.
  5. Uniform-PAC.
  6. Preliminaries
  7. Warmup: Frozen Policy Iteration in the PAC Setting
  8. The Algorithm
  9. Datasets.
  10. Exploration Mechanism.
  11. Avoiding Off-Policy Data and Resampling.
  12. Updating Datasets.
  13. Freezing High-Confidence States.
  14. The Analysis
  15. Time and Space Complexity.
  16. ...and 19 more sections

Key Result

Lemma 1

For any $t\ge 1,h\in[H]$, it holds that $|\mathcal{D}_{t,h}|\le D$.

Figures (2)

  • Figure 1: An example of MDP satisfying linear $Q^{\pi}$ realizability, where $\phi(s_1,a_1)=\phi(s_1,a_2)=e_1$, $\phi(s_2,a_3)=e_2$, $\phi(s_3,a_3)=e_3$.
  • Figure 2: Learning curves on CartPole-v1 and InvertedPendulum-v4. Results are averaged over 5 seeds. The shaded area captures a 95% confidence interval around the average performance.

Theorems & Definitions (57)

  • Lemma 1
  • Lemma 2
  • Lemma 3
  • Lemma 4
  • Lemma 5
  • Lemma 6
  • Theorem 1
  • Theorem 2
  • Lemma 7
  • proof
  • ...and 47 more