Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

End-to-End Efficient RL for Linear Bellman Complete MDPs with Deterministic Transitions

Zakaria Mhammedi, Alexander Rakhlin, Nneka Okolo

Abstract

We study reinforcement learning (RL) with linear function approximation in Markov Decision Processes (MDPs) satisfying \emph{linear Bellman completeness} -- a fundamental setting where the Bellman backup of any linear value function remains linear. While statistically tractable, prior computationally efficient algorithms are either limited to small action spaces or require strong oracle assumptions over the feature space. We provide a computationally efficient algorithm for linear Bellman complete MDPs with \emph{deterministic transitions}, stochastic initial states, and stochastic rewards. For finite action spaces, our algorithm is end-to-end efficient; for large or infinite action spaces, we require only a standard argmax oracle over actions. Our algorithm learns an $\varepsilon$-optimal policy with sample and computational complexity polynomial in the horizon, feature dimension, and $1/\varepsilon$.

End-to-End Efficient RL for Linear Bellman Complete MDPs with Deterministic Transitions

Abstract

We study reinforcement learning (RL) with linear function approximation in Markov Decision Processes (MDPs) satisfying \emph{linear Bellman completeness} -- a fundamental setting where the Bellman backup of any linear value function remains linear. While statistically tractable, prior computationally efficient algorithms are either limited to small action spaces or require strong oracle assumptions over the feature space. We provide a computationally efficient algorithm for linear Bellman complete MDPs with \emph{deterministic transitions}, stochastic initial states, and stochastic rewards. For finite action spaces, our algorithm is end-to-end efficient; for large or infinite action spaces, we require only a standard argmax oracle over actions. Our algorithm learns an -optimal policy with sample and computational complexity polynomial in the horizon, feature dimension, and .
Paper Structure (86 sections, 32 theorems, 203 equations, 11 algorithms)

This paper contains 86 sections, 32 theorems, 203 equations, 11 algorithms.

Table of Contents

  1. Introduction
  2. Linear Bellman completeness.
  3. The computational challenge.
  4. Our setting and contributions.
  5. Organization.
  6. Preliminaries
  7. Boundedness assumptions.
  8. Action argmax oracle.
  9. Goal.
  10. Additional notation.
  11. Warm Up: An Efficient Algorithm Under Finite Actions
  12. Prior work and context.
  13. High-level approach.
  14. Barycentric spanner.
  15. Optimistic Constraint Propagation (OCP).
  16. ...and 71 more sections

Key Result

Lemma 1

Fix $\varepsilon, \delta \in (0,1)$ and $R > 0$. Consider a call to OCP (alg:ocp) with reward functions $r_{1:H}$, function class $\mathcal{Q} = \{Q_{1:H} : Q_h(x,a) = \theta^\top\phi_h(x,a), \theta \in \mathbb{R}^d, |Q_h(x,a)| \le HR\}$, and number of episodes $T = \mathrm{poly}(|\mathcal{A}|, d, H Moreover, the computational cost of OCP is $\mathrm{poly}(|\mathcal{A}|, d, H, R, T)$.

Theorems & Definitions (33)

  • Definition 1: Linear Bellman completeness
  • Lemma 1: Guarantee of OCP
  • Lemma 2: Guarantee of $\textsc{ComputeSpanner}_{\texttt{ocp}}$
  • Theorem 1: End-to-end guarantee for OCP-based approach
  • Lemma 3: Guarantee of $\textsc{ComputeSpanner}_{\texttt{fqi}}$
  • Theorem 2: End-to-end suboptimality guarantee
  • Lemma 4: Validity of OCP as $\textsc{LinOpt}\xspace$
  • Lemma 5: Guarantee of RobustSpanner for $\textsc{ComputeSpanner}_{\texttt{ocp}}$
  • Lemma 6: Guarantee of $\textsc{PolicyOpt}_{\texttt{ocp}}$: Phase II
  • Lemma 7: Discovery of a new direction
  • ...and 23 more