Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Distributionally Robust Off-Dynamics Reinforcement Learning: Provable Efficiency with Linear Function Approximation

Zhishuai Liu, Pan Xu

TL;DR

The paper addresses off-dynamics reinforcement learning by formulating online distributionally robust MDPs with linear function approximation to bridge the sim-to-real gap. It shows that duals under TV-based uncertainty preserve linearity in the robust Q-function within a $d$-rectangular framework, mitigating nonlinearity and error amplification observed under KL/Chi-square divergences. The authors introduce the model-free DR-LSVI-UCB algorithm and prove a non-asymptotic suboptimality bound on the average suboptimality that scales as $\tilde{O}(\sqrt{H^4 d^4 / K})$, independent of the state and action space sizes, and demonstrate robustness through simulations including simulated linear MDPs and a American put option problem. This work advances theoretical understanding and practical viability of provably efficient online DRMDPs with linear function approximation for robust off-dynamics RL.

Abstract

We study off-dynamics Reinforcement Learning (RL), where the policy is trained on a source domain and deployed to a distinct target domain. We aim to solve this problem via online distributionally robust Markov decision processes (DRMDPs), where the learning algorithm actively interacts with the source domain while seeking the optimal performance under the worst possible dynamics that is within an uncertainty set of the source domain's transition kernel. We provide the first study on online DRMDPs with function approximation for off-dynamics RL. We find that DRMDPs' dual formulation can induce nonlinearity, even when the nominal transition kernel is linear, leading to error propagation. By designing a $d$-rectangular uncertainty set using the total variation distance, we remove this additional nonlinearity and bypass the error propagation. We then introduce DR-LSVI-UCB, the first provably efficient online DRMDP algorithm for off-dynamics RL with function approximation, and establish a polynomial suboptimality bound that is independent of the state and action space sizes. Our work makes the first step towards a deeper understanding of the provable efficiency of online DRMDPs with linear function approximation. Finally, we substantiate the performance and robustness of DR-LSVI-UCB through different numerical experiments.

Distributionally Robust Off-Dynamics Reinforcement Learning: Provable Efficiency with Linear Function Approximation

TL;DR

The paper addresses off-dynamics reinforcement learning by formulating online distributionally robust MDPs with linear function approximation to bridge the sim-to-real gap. It shows that duals under TV-based uncertainty preserve linearity in the robust Q-function within a -rectangular framework, mitigating nonlinearity and error amplification observed under KL/Chi-square divergences. The authors introduce the model-free DR-LSVI-UCB algorithm and prove a non-asymptotic suboptimality bound on the average suboptimality that scales as , independent of the state and action space sizes, and demonstrate robustness through simulations including simulated linear MDPs and a American put option problem. This work advances theoretical understanding and practical viability of provably efficient online DRMDPs with linear function approximation for robust off-dynamics RL.

Abstract

We study off-dynamics Reinforcement Learning (RL), where the policy is trained on a source domain and deployed to a distinct target domain. We aim to solve this problem via online distributionally robust Markov decision processes (DRMDPs), where the learning algorithm actively interacts with the source domain while seeking the optimal performance under the worst possible dynamics that is within an uncertainty set of the source domain's transition kernel. We provide the first study on online DRMDPs with function approximation for off-dynamics RL. We find that DRMDPs' dual formulation can induce nonlinearity, even when the nominal transition kernel is linear, leading to error propagation. By designing a -rectangular uncertainty set using the total variation distance, we remove this additional nonlinearity and bypass the error propagation. We then introduce DR-LSVI-UCB, the first provably efficient online DRMDP algorithm for off-dynamics RL with function approximation, and establish a polynomial suboptimality bound that is independent of the state and action space sizes. Our work makes the first step towards a deeper understanding of the provable efficiency of online DRMDPs with linear function approximation. Finally, we substantiate the performance and robustness of DR-LSVI-UCB through different numerical experiments.
Paper Structure (39 sections, 19 theorems, 115 equations, 4 figures, 1 algorithm)

This paper contains 39 sections, 19 theorems, 115 equations, 4 figures, 1 algorithm.

Table of Contents

  1. INTRODUCTION
  2. RELATED WORK
  3. Episodic Linear MDP
  4. DRMDPs
  5. DRMDPs with linear function approximation
  6. DISTRIBUTIONALLY ROBUST MDP WITH LINEAR FUNCTION APPROXIMATION
  7. Preliminaries
  8. $d$-Rectangular Linear DRMDP
  9. Robust Bellman Equation and the Optimal Policy in DRMDPs
  10. DRMDPs with TV divergence
  11. ROBUST LEAST SQUARE VALUE ITERATION WITH UCB EXPLORATION
  12. Linear Representation of the Robust State-Action Value Function
  13. UCB Exploration in DRMDP
  14. MAIN THEORETICAL RESULTS
  15. EXPERIMENTS
  16. ...and 24 more sections

Key Result

Proposition 3.2

(Robust Bellman equation) Under the $d$-rectangular linear DRMDP setting, for any nominal transition kernel $P^0=\{P^0_h\}_{h=1}^H$ and any stationary policy $\pi=\{\pi_h\}_{h=1}^H$, the following robust Bellman equation holds: for any $(h,s,a) \in [H]\times {\mathcal{S}} \times \mathcal{A}$,

Figures (4)

  • Figure 1: Simulation results under different source domains. The $x$-axis represents the perturbation level corresponding to different target environments. $\rho_{1,4}$ is the uncertainty level in our DR-LSVI-UCB algorithm.
  • Figure 2: Results for the simulated American put option problem. $\rho$ is the uncertainty level in DR-LSVI-UCB.
  • Figure 3: The source and the target linear MDP environments. The value on each arrow represents the transition probability. For the source MDP, there are five states and three steps, with the initial state being $x_1$, the fail state being $x_4$, and $x_5$ being an absorbing state with reward 1. The target MDP on the right is obtained by perturbing the transition probability at the first step of the source MDP, with others remaining the same.
  • Figure 4: Simulation results under different source domains. The $x$-axis represents the perturbation level corresponding to different target environments. $\rho_{1,4}$ is the input uncertainty level for our DR-LSVI-UCB algorithm.

Theorems & Definitions (25)

  • Proposition 3.2
  • Proposition 3.3
  • Proposition 3.4
  • Remark 4.2
  • Proposition 4.3
  • Remark 4.4
  • Remark 4.5
  • Remark 4.6
  • Theorem 5.1
  • Corollary 5.2
  • ...and 15 more