Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Optimistic Transfer under Task Shift via Bellman Alignment

Jinhang Chai, Enpei Zhang, Elynn Chen, Yujun Yan

TL;DR

This work addresses online transfer reinforcement learning where source-task data must be reused without compromising exploration. It identifies one-step Bellman mismatch as the core transfer obstacle and introduces re-weighted targeting (RWT) to align Bellman updates by using a density-ratio correction, reducing mismatch to a fixed one-step reward difference. The authors propose a two-stage RWT Q-learning framework that decouples variance reduction from structured bias correction and prove regret bounds under RKHS assumptions that depend on the task shift rather than the target MDP. Empirically, RWT improves sample efficiency in both tabular and neural network settings, confirming Bellman alignment as a model-agnostic principle for online transfer in RL. This approach provides principled guarantees for transfer under exploration and offers practical gains for simulators, personalization, and cross-domain decision making.

Abstract

We study online transfer reinforcement learning (RL) in episodic Markov decision processes, where experience from related source tasks is available during learning on a target task. A fundamental difficulty is that task similarity is typically defined in terms of rewards or transitions, whereas online RL algorithms operate on Bellman regression targets. As a result, naively reusing source Bellman updates introduces systematic bias and invalidates regret guarantees. We identify one-step Bellman alignment as the correct abstraction for transfer in online RL and propose re-weighted targeting (RWT), an operator-level correction that retargets continuation values and compensates for transition mismatch via a change of measure. RWT reduces task mismatch to a fixed one-step correction and enables statistically sound reuse of source data. This alignment yields a two-stage RWT $Q$-learning framework that separates variance reduction from bias correction. Under RKHS function approximation, we establish regret bounds that scale with the complexity of the task shift rather than the target MDP. Empirical results in both tabular and neural network settings demonstrate consistent improvements over single-task learning and naïve pooling, highlighting Bellman alignment as a model-agnostic transfer principle for online RL.

Optimistic Transfer under Task Shift via Bellman Alignment

TL;DR

This work addresses online transfer reinforcement learning where source-task data must be reused without compromising exploration. It identifies one-step Bellman mismatch as the core transfer obstacle and introduces re-weighted targeting (RWT) to align Bellman updates by using a density-ratio correction, reducing mismatch to a fixed one-step reward difference. The authors propose a two-stage RWT Q-learning framework that decouples variance reduction from structured bias correction and prove regret bounds under RKHS assumptions that depend on the task shift rather than the target MDP. Empirically, RWT improves sample efficiency in both tabular and neural network settings, confirming Bellman alignment as a model-agnostic principle for online transfer in RL. This approach provides principled guarantees for transfer under exploration and offers practical gains for simulators, personalization, and cross-domain decision making.

Abstract

We study online transfer reinforcement learning (RL) in episodic Markov decision processes, where experience from related source tasks is available during learning on a target task. A fundamental difficulty is that task similarity is typically defined in terms of rewards or transitions, whereas online RL algorithms operate on Bellman regression targets. As a result, naively reusing source Bellman updates introduces systematic bias and invalidates regret guarantees. We identify one-step Bellman alignment as the correct abstraction for transfer in online RL and propose re-weighted targeting (RWT), an operator-level correction that retargets continuation values and compensates for transition mismatch via a change of measure. RWT reduces task mismatch to a fixed one-step correction and enables statistically sound reuse of source data. This alignment yields a two-stage RWT -learning framework that separates variance reduction from bias correction. Under RKHS function approximation, we establish regret bounds that scale with the complexity of the task shift rather than the target MDP. Empirical results in both tabular and neural network settings demonstrate consistent improvements over single-task learning and naïve pooling, highlighting Bellman alignment as a model-agnostic transfer principle for online RL.
Paper Structure (41 sections, 3 theorems, 83 equations, 1 figure, 1 algorithm)

This paper contains 41 sections, 3 theorems, 83 equations, 1 figure, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work and Our Distinction
  3. Transferability via Bellman Alignment
  4. Problem Setup
  5. Why Naïve Bellman Transfer Fails
  6. Re-Weighted Targeting for Bellman Alignment
  7. Transferability via Structured Reward Differences
  8. RWT $Q$-Learning: Algorithmic Framework
  9. Synchronous Bellman Updates with RWT
  10. Exploration and Policy Deployment
  11. Algorithm Summary
  12. RKHS Instantiation and Regret Analysis
  13. RKHS Modeling of Structured Task Shift
  14. OFU-RWT $Q$-Learning under RKHS Approximation
  15. Regret Analysis
  16. ...and 26 more sections

Key Result

Theorem 4.6

Under the baseline and correction RKHS complexity conditions (Definition definition:rkhs-complexity-source--definition:rkhs-complexity-correction), together with source coverage (Assumption assume:source-coverage) and density-ratio estimation (Assumption assume:density-ratio), with probability at le

Figures (1)

  • Figure 1: Learning curves comparing RWT-$Q$, naïve pooled $Q$-learning, and target-only $Q$-learning. Top: RandomRewardGridEnv with tabular $Q$-learning. Bottom: RandomRewardGridEnv with DQN function approximation. RWT-$Q$ consistently improves sample efficiency, while naïve pooling often degrades performance due to Bellman misalignment.

Theorems & Definitions (7)

  • Definition 4.2: Baseline RKHS complexity (source stage)
  • Definition 4.3: Correction RKHS complexity (target stage)
  • Theorem 4.6: Regret under RKHS Instantiation
  • Lemma B.1
  • proof
  • Lemma B.2: Deterministic self-normalized bound
  • proof