Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Sample Complexity of Distributionally Robust Off-Dynamics Reinforcement Learning with Online Interaction

Yiting He, Zhishuai Liu, Weixin Wang, Pan Xu

TL;DR

This work analyzes online reinforcement learning in distributionally robust finite-horizon MDPs, introducing the supremal visitation ratio C_vr as a key measure of exploration difficulty. It proposes ORBIT, a computationally efficient online algorithm that achieves sublinear regret for CRMDPs and RRMDPs under general f-divergence based uncertainties (TV, KL, Chi-squared) and provides matching lower bounds to show optimal dependence on C_vr and the number of episodes. Theoretical results reveal that bounded C_vr enables provably efficient online learning while unbounded C_vr can cause exponential hardness, and the empirical studies on simulated MDPs and Frozen Lake corroborate the theory and demonstrate robustness under distribution shifts.

Abstract

Off-dynamics reinforcement learning (RL), where training and deployment transition dynamics are different, can be formulated as learning in a robust Markov decision process (RMDP) where uncertainties in transition dynamics are imposed. Existing literature mostly assumes access to generative models allowing arbitrary state-action queries or pre-collected datasets with a good state coverage of the deployment environment, bypassing the challenge of exploration. In this work, we study a more realistic and challenging setting where the agent is limited to online interaction with the training environment. To capture the intrinsic difficulty of exploration in online RMDPs, we introduce the supremal visitation ratio, a novel quantity that measures the mismatch between the training dynamics and the deployment dynamics. We show that if this ratio is unbounded, online learning becomes exponentially hard. We propose the first computationally efficient algorithm that achieves sublinear regret in online RMDPs with $f$-divergence based transition uncertainties. We also establish matching regret lower bounds, demonstrating that our algorithm achieves optimal dependence on both the supremal visitation ratio and the number of interaction episodes. Finally, we validate our theoretical results through comprehensive numerical experiments.

Sample Complexity of Distributionally Robust Off-Dynamics Reinforcement Learning with Online Interaction

TL;DR

This work analyzes online reinforcement learning in distributionally robust finite-horizon MDPs, introducing the supremal visitation ratio C_vr as a key measure of exploration difficulty. It proposes ORBIT, a computationally efficient online algorithm that achieves sublinear regret for CRMDPs and RRMDPs under general f-divergence based uncertainties (TV, KL, Chi-squared) and provides matching lower bounds to show optimal dependence on C_vr and the number of episodes. Theoretical results reveal that bounded C_vr enables provably efficient online learning while unbounded C_vr can cause exponential hardness, and the empirical studies on simulated MDPs and Frozen Lake corroborate the theory and demonstrate robustness under distribution shifts.

Abstract

Off-dynamics reinforcement learning (RL), where training and deployment transition dynamics are different, can be formulated as learning in a robust Markov decision process (RMDP) where uncertainties in transition dynamics are imposed. Existing literature mostly assumes access to generative models allowing arbitrary state-action queries or pre-collected datasets with a good state coverage of the deployment environment, bypassing the challenge of exploration. In this work, we study a more realistic and challenging setting where the agent is limited to online interaction with the training environment. To capture the intrinsic difficulty of exploration in online RMDPs, we introduce the supremal visitation ratio, a novel quantity that measures the mismatch between the training dynamics and the deployment dynamics. We show that if this ratio is unbounded, online learning becomes exponentially hard. We propose the first computationally efficient algorithm that achieves sublinear regret in online RMDPs with -divergence based transition uncertainties. We also establish matching regret lower bounds, demonstrating that our algorithm achieves optimal dependence on both the supremal visitation ratio and the number of interaction episodes. Finally, we validate our theoretical results through comprehensive numerical experiments.

Paper Structure

This paper contains 61 sections, 47 theorems, 203 equations, 6 figures, 3 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Notations
  3. Related Work
  4. Preliminaries
  5. Constrained Robust MDP (CRMDP)
  6. Regularized Robust MDP (RRMDP)
  7. Learning Goal
  8. Online Robust Bellman Iteration (ORBIT)
  9. Algorithm Interpretation
  10. ORBIT under Constrained Robust MDPs
  11. CRMDP-TV
  12. CRMDP-KL
  13. CRMDP-$\chi^2$
  14. ORBIT under Regularized Robust MDPs
  15. RRMDP-TV
  16. ...and 46 more sections

Key Result

Proposition 5.4

For CRMDPs with TV-distance defined uncertainty set satisfying con:fail-states, for any $s\in\mathcal{S},\,a\in\mathcal{A},\,s'\in\mathcal{S}\backslash\mathcal{S}_f$ and policy $\pi$, we have $P_h^{w,\pi}(s'|s,a)\leq P_h^o(s'|s,a)$.

Figures (6)

  • Figure 1: \ref{['fig:visit ratio result']} shows the comparison of the learned policy and the optimal policy in \ref{['sec:main show visit ratio']} (Illustration of the Effect of $C_{vr}$ on Robustness), where the optimal policy represents the ground truth optimal policy, the learned policy is obtained by \ref{['alg:main alg']}. \ref{['fig:Simple Example TV', 'fig:Simple Example KL', 'fig:Simple Example Chi2']} present the comparison between our algorithm ORBIT and the non-robust algorithm in \ref{['sec:main Simple Example']} (Learning on Simulated RMDPs).
  • Figure 2: The convergence of ORBIT in \ref{['sec:main Frozen Lake']} (Learning the Frozen Lake Problem). We use the policies obtained after each training episode to evaluate the convergence.
  • Figure 3: Results in \ref{['sec:main Frozen Lake']} (Learning the Frozen Lake Problem). \ref{['fig:exp Lake time']} presents the average time taken for training in various settings. \ref{['fig:exp Lake TV res', 'fig:exp Lake KL res', 'fig:exp Lake Chi2 res']} present the comparison between our algorithm ORBIT and the non-robust algorithm. We use the last episode policy $\pi^K$ for the comparison.
  • Figure 4: The constructions of the nominal MDP and the worst-case MDP environments in \ref{['sec:main show visit ratio']}.
  • Figure 5: The source and target RMDP environments in \ref{['sec:main Simple Example']}, where the target environment is constructed by perturbing the first step.
  • ...and 1 more figures

Theorems & Definitions (56)

  • Definition 5.3: Visitation measure
  • Proposition 5.4
  • Remark 5.6: Reduction to non-robust setting
  • Remark 5.8
  • Theorem 5.9: CRMDP regret upper bounds
  • Remark 5.10
  • Corollary 5.11
  • Theorem 5.12: CRMDP regret lower bound
  • Remark 5.13
  • Lemma 5.14: CRMDP hard instances
  • ...and 46 more