Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Learning Upper Lower Value Envelopes to Shape Online RL: A Principled Approach

Sebastian Reboul, Hélène Halconruy, Randal Douc

TL;DR

The paper tackles the challenge of using offline data to speed up online reinforcement learning by learning upper and lower value envelopes (value envelopes) that bound the optimal value and action-value functions. It introduces a principled two-stage approach: an offline phase learns envelope functions $(W_h,U_h,\widetilde{W}_h,\widetilde{U}_h)$ that bound $Q_h^*$ and $V_h^*$ with high probability, and an online phase uses these random envelopes to shape exploration in a UCBVI-like algorithm, via either $Q$-shaping or $V$-shaping updates and a variance-aware bonus. Theoretical results establish high-probability regret bounds that depend on the envelope width $D^{\max}$ and envelope range $R^{\max}$, and show how regret decreases as offline data improves envelope accuracy (e.g., $D^{\max}$ tends to zero with large $K$). Empirical experiments on layered tabular MDPs demonstrate substantial improvements over standard UCBVI and prior shaping-based methods, illustrating a formal bridge between offline pre-training and online fine-tuning. The framework is extensible to other RL settings and suggests promising directions for transfer learning and efficient offline-online synthesis of shaping information.

Abstract

We investigate the fundamental problem of leveraging offline data to accelerate online reinforcement learning - a direction with strong potential but limited theoretical grounding. Our study centers on how to learn and apply value envelopes within this context. To this end, we introduce a principled two-stage framework: the first stage uses offline data to derive upper and lower bounds on value functions, while the second incorporates these learned bounds into online algorithms. Our method extends prior work by decoupling the upper and lower bounds, enabling more flexible and tighter approximations. In contrast to approaches that rely on fixed shaping functions, our envelopes are data-driven and explicitly modeled as random variables, with a filtration argument ensuring independence across phases. The analysis establishes high-probability regret bounds determined by two interpretable quantities, thereby providing a formal bridge between offline pre-training and online fine-tuning. Empirical results on tabular MDPs demonstrate substantial regret reductions compared with both UCBVI and prior methods.

Learning Upper Lower Value Envelopes to Shape Online RL: A Principled Approach

TL;DR

The paper tackles the challenge of using offline data to speed up online reinforcement learning by learning upper and lower value envelopes (value envelopes) that bound the optimal value and action-value functions. It introduces a principled two-stage approach: an offline phase learns envelope functions that bound and with high probability, and an online phase uses these random envelopes to shape exploration in a UCBVI-like algorithm, via either -shaping or -shaping updates and a variance-aware bonus. Theoretical results establish high-probability regret bounds that depend on the envelope width and envelope range , and show how regret decreases as offline data improves envelope accuracy (e.g., tends to zero with large ). Empirical experiments on layered tabular MDPs demonstrate substantial improvements over standard UCBVI and prior shaping-based methods, illustrating a formal bridge between offline pre-training and online fine-tuning. The framework is extensible to other RL settings and suggests promising directions for transfer learning and efficient offline-online synthesis of shaping information.

Abstract

We investigate the fundamental problem of leveraging offline data to accelerate online reinforcement learning - a direction with strong potential but limited theoretical grounding. Our study centers on how to learn and apply value envelopes within this context. To this end, we introduce a principled two-stage framework: the first stage uses offline data to derive upper and lower bounds on value functions, while the second incorporates these learned bounds into online algorithms. Our method extends prior work by decoupling the upper and lower bounds, enabling more flexible and tighter approximations. In contrast to approaches that rely on fixed shaping functions, our envelopes are data-driven and explicitly modeled as random variables, with a filtration argument ensuring independence across phases. The analysis establishes high-probability regret bounds determined by two interpretable quantities, thereby providing a formal bridge between offline pre-training and online fine-tuning. Empirical results on tabular MDPs demonstrate substantial regret reductions compared with both UCBVI and prior methods.
Paper Structure (48 sections, 24 theorems, 233 equations, 5 figures, 3 algorithms)

This paper contains 48 sections, 24 theorems, 233 equations, 5 figures, 3 algorithms.

Table of Contents

  1. Introduction
  2. Motivation
  3. Contributions
  4. Organization of the paper
  5. Related work
  6. Gap-dependent regret bounds
  7. Prior knowledge
  8. Accelerating online RL with offline knowledge
  9. Reward shaping
  10. Fine-tuning
  11. Statistical setting
  12. Markov decision process
  13. Offline algorithms (see li_settling_2024)
  14. Online algorithms (see zhang2024settling)
  15. Method: Envelope Shaping
  16. ...and 33 more sections

Key Result

Theorem 1

Suppose that Assumption assumption:sandwich holds with some constant $\delta>0$. Then, with probability at least $1-4\delta$, the regret of Algorithm alg:q_shaping after $T$ episodes satisfies: where $\widetilde{\mathcal{O}}$ hides polylog factors in $T,|\mathcal{S}|,|\mathcal{A}|,H,$ and $1/\delta$, and $R^{\max}$ and $D^{\max}$ are defined by eq:D_R_max, respectively and

Figures (5)

  • Figure 1: Upper and lower envelopes $U_{h}$ and $W_{h}$ around $V_h^\star$. The vertical indicators $D_h(s_i)$ show the varying gap width between the envelopes at specific points.
  • Figure 2: Illustration of how $D_h {\to} 0$ and $R_h{\to} \mathrm{Range}(V^{\star}_h)$ as $K {\to} \infty$. Here there are 3 actions, $H{=}4$ with 3 states per layer. The plot represents $D_2(s)$ for the states in layer $h=2$ with respect to $R_h$ for varying $K$.
  • Figure 3: Layered MDP with $H {=} 4$ and 3 states per layer with 3 actions. Average over $4$ seeds.
  • Figure 4: Expanding Range
  • Figure 5: Sliding Range

Theorems & Definitions (43)

  • Remark 1
  • Remark 2
  • Theorem 1
  • Theorem 2
  • Theorem 3: $V$-shaping Regret
  • Definition 1: Finite-Horizon MDP
  • Definition 2: Episode Filtration
  • Remark 3
  • Lemma 1: Offline Bonus Validity
  • proof
  • ...and 33 more