Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Linear Mixture Distributionally Robust Markov Decision Processes

Zhishuai Liu, Pan Xu

TL;DR

This work introduces the linear mixture distributionally robust MDP (DRMDP) framework to address off-dynamics RL where target transitions deviate from a known source via perturbations in mixture weights. By modeling $P_h^0(\cdot|s,a)=\boldsymbol{\phi}(\cdot|s,a)^T \boldsymbol{\theta}_h^0$ and constraining $\boldsymbol{\theta}_h$ within a divergence-based neighborhood, the authors derive a robust Bellman equation and establish DP principles under this structured uncertainty. They propose a meta algorithm with double pessimism and present two practical algorithms, DRTTR and DRVTR, based on transition-targeted and value-targeted regression, respectively, along with finite-sample guarantees under TV, KL, and $\chi^2$ divergences. Simulations demonstrate that the linear mixture DRMDP can yield less conservative, more robust policies than standard rectangular uncertainty sets, validating the approach for offline robustness and guiding future extensions to online settings and model misspecification analysis.

Abstract

Many real-world decision-making problems face the off-dynamics challenge: the agent learns a policy in a source domain and deploys it in a target domain with different state transitions. The distributionally robust Markov decision process (DRMDP) addresses this challenge by finding a robust policy that performs well under the worst-case environment within a pre-specified uncertainty set of transition dynamics. Its effectiveness heavily hinges on the proper design of these uncertainty sets, based on prior knowledge of the dynamics. In this work, we propose a novel linear mixture DRMDP framework, where the nominal dynamics is assumed to be a linear mixture model. In contrast with existing uncertainty sets directly defined as a ball centered around the nominal kernel, linear mixture DRMDPs define the uncertainty sets based on a ball around the mixture weighting parameter. We show that this new framework provides a more refined representation of uncertainties compared to conventional models based on $(s,a)$-rectangularity and $d$-rectangularity, when prior knowledge about the mixture model is present. We propose a meta algorithm for robust policy learning in linear mixture DRMDPs with general $f$-divergence defined uncertainty sets, and analyze its sample complexities under three divergence metrics instantiations: total variation, Kullback-Leibler, and $χ^2$ divergences. These results establish the statistical learnability of linear mixture DRMDPs, laying the theoretical foundation for future research on this new setting.

Linear Mixture Distributionally Robust Markov Decision Processes

TL;DR

This work introduces the linear mixture distributionally robust MDP (DRMDP) framework to address off-dynamics RL where target transitions deviate from a known source via perturbations in mixture weights. By modeling and constraining within a divergence-based neighborhood, the authors derive a robust Bellman equation and establish DP principles under this structured uncertainty. They propose a meta algorithm with double pessimism and present two practical algorithms, DRTTR and DRVTR, based on transition-targeted and value-targeted regression, respectively, along with finite-sample guarantees under TV, KL, and divergences. Simulations demonstrate that the linear mixture DRMDP can yield less conservative, more robust policies than standard rectangular uncertainty sets, validating the approach for offline robustness and guiding future extensions to online settings and model misspecification analysis.

Abstract

Many real-world decision-making problems face the off-dynamics challenge: the agent learns a policy in a source domain and deploys it in a target domain with different state transitions. The distributionally robust Markov decision process (DRMDP) addresses this challenge by finding a robust policy that performs well under the worst-case environment within a pre-specified uncertainty set of transition dynamics. Its effectiveness heavily hinges on the proper design of these uncertainty sets, based on prior knowledge of the dynamics. In this work, we propose a novel linear mixture DRMDP framework, where the nominal dynamics is assumed to be a linear mixture model. In contrast with existing uncertainty sets directly defined as a ball centered around the nominal kernel, linear mixture DRMDPs define the uncertainty sets based on a ball around the mixture weighting parameter. We show that this new framework provides a more refined representation of uncertainties compared to conventional models based on -rectangularity and -rectangularity, when prior knowledge about the mixture model is present. We propose a meta algorithm for robust policy learning in linear mixture DRMDPs with general -divergence defined uncertainty sets, and analyze its sample complexities under three divergence metrics instantiations: total variation, Kullback-Leibler, and divergences. These results establish the statistical learnability of linear mixture DRMDPs, laying the theoretical foundation for future research on this new setting.

Paper Structure

This paper contains 27 sections, 14 theorems, 68 equations, 7 figures, 2 algorithms.

Table of Contents

  1. Introduction
  2. Related works
  3. Linear mixture distributionally robust Markov decision process
  4. Offline dataset and learning goal
  5. Comparison with $(s,a)$-rectangular uncertainty sets
  6. Comparison with $d$-rectangular uncertainty sets
  7. Algorithms and theoretical guarantees
  8. TV-divergence
  9. KL-divergence
  10. $\chi^2$-divergence
  11. Practical algorithms
  12. Transition-targeted regression
  13. Value-targeted regression
  14. Simulation study
  15. Experiment setup
  16. ...and 12 more sections

Key Result

Proposition 3.2

Under the linear mixture DRMDP setting, for any nominal transition kernel $P^0$ and any stationary policy $\pi=\{\pi_h\}_{h=1}^H$, the following robust Bellman equation holds: for any $(h,s,a)\in[H]\times{\mathcal{S}}\times\mathcal{A}$,

Figures (7)

  • Figure 1: An illustration of the linear mixture uncertainty set and the standard $(s,a)$-rectangular uncertainty set in $\mathbb{R}^3$. ${\mathcal{S}}=\{x_1, x_2, x_3\}$. The yellow region represents the probability simplex and each point in the region is a probability distribution. $\phi_1$ and $\phi_2$ are two basis modes, $\bm{\theta}=[1/2, 1/2]^{\top}$, the nominal kernel is $P^0$. The linear mixture uncertainty set with radius $\rho=1/8$ is the black segment. The smallest $(s,a)$-rectangular uncertainty set covering the linear mixture uncertainty set is the orange octagon centered around $P^0$ with radius 0.1.
  • Figure 2: An illustration of the linear mixture uncertainty set and the $d$-rectangular uncertainty set.
  • Figure 3: The source and the target linear MDP environments. The value on each arrow represents the transition probability. For the source MDP, there are five states and three steps, with the initial state being $x_1$, the fail state being $x_4$, and $x_5$ being an absorbing state with reward 1. The target MDP on the right is obtained by perturbing the transition probability at the first step of the source MDP, with others remaining the same.
  • Figure 4: Simulation results of DRTTR under different source domains. Policies are learned from the nominal environment featuring $\bm{\theta}_1 = (0,0.9,0.1)$. Numbers in parenthesis represent $(\delta, \Vert\xi\Vert_1, \rho_{\text{TV}},\rho_{\text{KL}},\rho_{\chi^2})$, respectively. The $x$-axis represents the perturbation level corresponding to different target environments. $\rho_{\text{TV}},\rho_{\text{KL}}$ and $\rho_{\chi^2}$ are the input uncertainty levels for our DRTTR algorithm.
  • Figure 5: Simulation results of DRVTR under different source domains. Policies are learned from the nominal environment featuring $\bm{\theta}_1 = (0,0.9,0.1)$. The $x$-axis represents the perturbation level corresponding to different target environments. $\rho_{\text{TV}},\rho_{\text{KL}}$ and $\rho_{\chi^2}$ are the input uncertainty levels for our DRVTR algorithm.
  • ...and 2 more figures

Theorems & Definitions (22)

  • Proposition 3.2: Robust Bellman Equation
  • Proposition 3.3: Existence of the optimal policy
  • Lemma 3.4
  • Lemma 3.5
  • Remark 3.6
  • Remark 4.2
  • Lemma 4.3
  • Remark 4.5
  • Theorem 4.6: TV-divergence
  • Remark 4.8
  • ...and 12 more