Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Near-Optimal Regret for Policy Optimization in Contextual MDPs with General Offline Function Approximation

Orin Levy, Aviv Rosenberg, Alon Cohen, Yishay Mansour

TL;DR

The first regret bound with optimal dependence on S and A is introduced, directly improving the current state-of-the-art CMDPs and demonstrating that optimistic policy optimization provides a natural, computationally superior and theoretically near-optimal path for solving CMDPs.

Abstract

We introduce \texttt{OPO-CMDP}, the first policy optimization algorithm for stochastic Contextual Markov Decision Process (CMDPs) under general offline function approximation. Our approach achieves a high probability regret bound of $\widetilde{O}(H^4\sqrt{T|S||A|\log(|\mathcal{F}||\mathcal{P}|)}),$ where $S$ and $A$ denote the state and action spaces, $H$ the horizon length, $T$ the number of episodes, and $\mathcal{F}, \mathcal{P}$ the finite function classes used to approximate the losses and dynamics, respectively. This is the first regret bound with optimal dependence on $|S|$ and $|A|$, directly improving the current state-of-the-art (Qian, Hu, and Simchi-Levi, 2024). These results demonstrate that optimistic policy optimization provides a natural, computationally superior and theoretically near-optimal path for solving CMDPs.

Near-Optimal Regret for Policy Optimization in Contextual MDPs with General Offline Function Approximation

TL;DR

The first regret bound with optimal dependence on S and A is introduced, directly improving the current state-of-the-art CMDPs and demonstrating that optimistic policy optimization provides a natural, computationally superior and theoretically near-optimal path for solving CMDPs.

Abstract

We introduce \texttt{OPO-CMDP}, the first policy optimization algorithm for stochastic Contextual Markov Decision Process (CMDPs) under general offline function approximation. Our approach achieves a high probability regret bound of where and denote the state and action spaces, the horizon length, the number of episodes, and the finite function classes used to approximate the losses and dynamics, respectively. This is the first regret bound with optimal dependence on and , directly improving the current state-of-the-art (Qian, Hu, and Simchi-Levi, 2024). These results demonstrate that optimistic policy optimization provides a natural, computationally superior and theoretically near-optimal path for solving CMDPs.
Paper Structure (20 sections, 28 theorems, 91 equations, 1 table, 1 algorithm)

This paper contains 20 sections, 28 theorems, 91 equations, 1 table, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Literature
  3. Preliminaries
  4. Episodic Loop-Free MDP
  5. Stochastic Contextual MDP (CMDP)
  6. Offline Function Approximation
  7. Main Result: Provable Policy Optimization Algorithm for CMDPs
  8. Regret Analysis
  9. Proof Outline
  10. Main Technical Novelty: Deriving the Bonus Bound
  11. Discussion and Conclusion
  12. Proofs
  13. Deriving The Value-Confidence Bounds
  14. Deriving the Bonus Bound
  15. Driving the Final Regret Bound
  16. ...and 5 more sections

Key Result

Corollary 2.1

Let $\hat{f}^t \in \mathcal{F}$ be the least squares minimizer. For any $\delta \in (0,1)$, it holds with probability at least $1-\delta$, where $\mathcal{E}^t_{\mathrm{sq}}(\pi,c)$ is the expected squared error at round $t$,

Theorems & Definitions (44)

  • Corollary 2.1: Corollary 4.2 in DBLP:conf/icml/LevyCCM24
  • Definition 2.2: Squared Hellinger Distance
  • Corollary 2.3: Corollary 4.3 in DBLP:conf/icml/LevyCCM24
  • Theorem 3.1
  • Lemma 4.1
  • Lemma 4.2
  • Lemma 4.3
  • Lemma 4.4
  • Claim 4.6
  • Lemma 4.7
  • ...and 34 more