Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Eluder-based Regret for Stochastic Contextual MDPs

Orin Levy, Asaf Cassel, Alon Cohen, Yishay Mansour

TL;DR

This work tackles regret minimization in stochastic Contextual MDPs (CMDPs) under offline function approximation. It introduces E-UC$^3$RL, an algorithm built on optimistic-in-expectation principles that leverages offline least-squares and log-loss regression oracles to learn context-dependent dynamics and rewards. The main theoretical contribution is a rate-optimal regret bound of order $\tilde{O}(H^3 \sqrt{T|S||A|d_{\mathrm{E}}(\mathcal{P}) \log\left(|\mathcal{F}|\,|\mathcal{P}|/\delta\right)})$, with $d_{\mathrm{E}}(\mathcal{P})$ the Eluder dimension in the squared Hellinger distance, making the method efficient under offline regression. A key technical advancement is extending the Eluder dimension to general bounded metrics and a multiplicative value-change-of-measure lemma that ties estimation errors to Hellinger-distance-based regret, enabling computable confidence bounds and high-probability regret guarantees. This work thus provides the first efficient, rate-optimal CMDP algorithm in the offline regression setting and introduces broader metric-Eluder tools for RL analysis.

Abstract

We present the E-UC$^3$RL algorithm for regret minimization in Stochastic Contextual Markov Decision Processes (CMDPs). The algorithm operates under the minimal assumptions of realizable function class and access to \emph{offline} least squares and log loss regression oracles. Our algorithm is efficient (assuming efficient offline regression oracles) and enjoys a regret guarantee of $ \widetilde{O}(H^3 \sqrt{T |S| |A|d_{\mathrm{E}}(\mathcal{P}) \log (|\mathcal{F}| |\mathcal{P}|/ δ) )}) , $ with $T$ being the number of episodes, $S$ the state space, $A$ the action space, $H$ the horizon, $\mathcal{P}$ and $\mathcal{F}$ are finite function classes used to approximate the context-dependent dynamics and rewards, respectively, and $d_{\mathrm{E}}(\mathcal{P})$ is the Eluder dimension of $\mathcal{P}$ w.r.t the Hellinger distance. To the best of our knowledge, our algorithm is the first efficient and rate-optimal regret minimization algorithm for CMDPs that operates under the general offline function approximation setting. In addition, we extend the Eluder dimension to general bounded metrics which may be of separate interest.

Eluder-based Regret for Stochastic Contextual MDPs

TL;DR

This work tackles regret minimization in stochastic Contextual MDPs (CMDPs) under offline function approximation. It introduces E-UCRL, an algorithm built on optimistic-in-expectation principles that leverages offline least-squares and log-loss regression oracles to learn context-dependent dynamics and rewards. The main theoretical contribution is a rate-optimal regret bound of order , with the Eluder dimension in the squared Hellinger distance, making the method efficient under offline regression. A key technical advancement is extending the Eluder dimension to general bounded metrics and a multiplicative value-change-of-measure lemma that ties estimation errors to Hellinger-distance-based regret, enabling computable confidence bounds and high-probability regret guarantees. This work thus provides the first efficient, rate-optimal CMDP algorithm in the offline regression setting and introduces broader metric-Eluder tools for RL analysis.

Abstract

We present the E-UCRL algorithm for regret minimization in Stochastic Contextual Markov Decision Processes (CMDPs). The algorithm operates under the minimal assumptions of realizable function class and access to \emph{offline} least squares and log loss regression oracles. Our algorithm is efficient (assuming efficient offline regression oracles) and enjoys a regret guarantee of with being the number of episodes, the state space, the action space, the horizon, and are finite function classes used to approximate the context-dependent dynamics and rewards, respectively, and is the Eluder dimension of w.r.t the Hellinger distance. To the best of our knowledge, our algorithm is the first efficient and rate-optimal regret minimization algorithm for CMDPs that operates under the general offline function approximation setting. In addition, we extend the Eluder dimension to general bounded metrics which may be of separate interest.
Paper Structure (20 sections, 36 theorems, 89 equations, 1 algorithm)

This paper contains 20 sections, 36 theorems, 89 equations, 1 algorithm.

Table of Contents

  1. Introduction
  2. Preliminaries
  3. Episodic Loop-Free Markov Decision Process (MDP)
  4. Problem Setup: Stochastic Contextual Markov Decision Process (CMDP)
  5. Metric Eluder Dimension
  6. Function Class Assumptions
  7. Algorithm and Main Result
  8. Analysis
  9. Discussion and Conclusion
  10. Metric Eluder Dimension
  11. Upper bound using the $\ell_2$ Eluder dimension
  12. Proofs
  13. Multiplicative Value Change of Measure
  14. Oracle Bounds (Step 1)
  15. Reward oracle.
  16. ...and 5 more sections

Key Result

Lemma 2.2

For any $t \in [T], h \in [H]$ let $x_h^t \in \mathcal{X}$ and ${P}_t \in \mathcal{P}$ be arbitrary. Define the confidence sets with parameter $\beta$ as We have that

Theorems & Definitions (56)

  • Definition 2.1: Metric-Eluder Dimension
  • Lemma 2.2
  • Definition 2.3: Squared Hellinger Distance
  • Theorem 3.1: E-UC$^3$RL regret bound
  • Lemma 4.1: Value change of measure
  • Corollary 4.2: Reward approximation bound
  • Corollary 4.3: Dynamics approximation bound
  • Lemma 4.4: Stability error of log-loss oracle
  • Lemma 4.5: Confidence bound for rewards approximation w.r.t the approximated dynamics
  • Lemma 4.6: Confidence bound for dynamics approximation w.r.t the true rewards $f_\star$
  • ...and 46 more