Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Near-Optimal Learning and Planning in Separated Latent MDPs

Fan Chen, Constantinos Daskalakis, Noah Golowich, Alexander Rakhlin

TL;DR

The main thrust of this paper is in establishing a nearly-sharp *statistical threshold* for the horizon length necessary for efficient learning in Latent Markov Decision Processes.

Abstract

We study computational and statistical aspects of learning Latent Markov Decision Processes (LMDPs). In this model, the learner interacts with an MDP drawn at the beginning of each epoch from an unknown mixture of MDPs. To sidestep known impossibility results, we consider several notions of separation of the constituent MDPs. The main thrust of this paper is in establishing a nearly-sharp *statistical threshold* for the horizon length necessary for efficient learning. On the computational side, we show that under a weaker assumption of separability under the optimal policy, there is a quasi-polynomial algorithm with time complexity scaling in terms of the statistical threshold. We further show a near-matching time complexity lower bound under the exponential time hypothesis.

Near-Optimal Learning and Planning in Separated Latent MDPs

TL;DR

The main thrust of this paper is in establishing a nearly-sharp *statistical threshold* for the horizon length necessary for efficient learning in Latent Markov Decision Processes.

Abstract

We study computational and statistical aspects of learning Latent Markov Decision Processes (LMDPs). In this model, the learner interacts with an MDP drawn at the beginning of each epoch from an unknown mixture of MDPs. To sidestep known impossibility results, we consider several notions of separation of the constituent MDPs. The main thrust of this paper is in establishing a nearly-sharp *statistical threshold* for the horizon length necessary for efficient learning. On the computational side, we show that under a weaker assumption of separability under the optimal policy, there is a quasi-polynomial algorithm with time complexity scaling in terms of the statistical threshold. We further show a near-matching time complexity lower bound under the exponential time hypothesis.
Paper Structure (76 sections, 57 theorems, 317 equations, 2 algorithms)

This paper contains 76 sections, 57 theorems, 317 equations, 2 algorithms.

Table of Contents

  1. Introduction
  2. Our Contributions.
  3. Related works
  4. Planning in partially observable environment.
  5. Learning in partially observable environment.
  6. RL with function approximation.
  7. Preliminaries
  8. Latent Markov Decision Process.
  9. Policies.
  10. Miscellaneous notations
  11. Strong separation and separation under policies
  12. Model-based function approximation
  13. Learning goal.
  14. Intractability of separated LMDP with horizon below threshold
  15. Proof idea for \ref{['thm:lower-demo']}.
  16. ...and 61 more sections

Key Result

Proposition 2.4

If the LMDP $M$ is $\delta$-strongly separated, then it is $\varpi_{\delta}$-separated under any policy $\pi\in\Pi_{\rm RND}$, where $\varpi_{\delta}(h)=\frac{\delta^2}{2}(h-1)$.

Theorems & Definitions (67)

  • Definition 2.1: Strong separation, kwon2021rl
  • Definition 2.2: Decodability, efroni2022provable
  • Definition 2.3: Separation with respect to a policy
  • Proposition 2.4
  • Proposition 2.5
  • Lemma 2.6
  • Theorem 3.1: Corollary of \ref{['thm:log-L', 'thm:log-eps']}
  • Theorem 3.2
  • Proposition 3.3: Simplified version of \ref{['prop:2-family']}
  • Theorem 3.4
  • ...and 57 more