Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Robust Offline Reinforcement Learning with Linearly Structured f-Divergence Regularization

Cheng Tang, Zhishuai Liu, Pan Xu

TL;DR

The paper addresses offline reinforcement learning under dynamics shifts by formulating a d-rectangular linear RRMDP that linearizes rewards and transitions via a simplex feature map. It introduces R2PVI, a scalable offline algorithm with closed-form duals for TV and KL divergences and a per-dimension approach for chi-squared divergence, enabling efficient robust policy learning. The authors provide instance-dependent and instance-independent suboptimality bounds and prove an information-theoretic lower bound, demonstrating near-optimality, complemented by experiments on simulated linear MDPs and an American Put Option to validate robustness and computational efficiency. This work advances robust offline RL by replacing traditional uncertainty sets with linearly structured regularization, offering practical benefits in high-dimensional settings while preserving strong theoretical guarantees.

Abstract

The Robust Regularized Markov Decision Process (RRMDP) is proposed to learn policies robust to dynamics shifts by adding regularization to the transition dynamics in the value function. Existing methods mostly use unstructured regularization, potentially leading to conservative policies under unrealistic transitions. To address this limitation, we propose a novel framework, the $d$-rectangular linear RRMDP ($d$-RRMDP), which introduces latent structures into both transition kernels and regularization. We focus on offline reinforcement learning, where an agent learns policies from a precollected dataset in the nominal environment. We develop the Robust Regularized Pessimistic Value Iteration (R2PVI) algorithm that employs linear function approximation for robust policy learning in $d$-RRMDPs with $f$-divergence based regularization terms on transition kernels. We provide instance-dependent upper bounds on the suboptimality gap of R2PVI policies, demonstrating that these bounds are influenced by how well the dataset covers state-action spaces visited by the optimal robust policy under robustly admissible transitions. We establish information-theoretic lower bounds to verify that our algorithm is near-optimal. Finally, numerical experiments validate that R2PVI learns robust policies and exhibits superior computational efficiency compared to baseline methods.

Robust Offline Reinforcement Learning with Linearly Structured f-Divergence Regularization

TL;DR

The paper addresses offline reinforcement learning under dynamics shifts by formulating a d-rectangular linear RRMDP that linearizes rewards and transitions via a simplex feature map. It introduces R2PVI, a scalable offline algorithm with closed-form duals for TV and KL divergences and a per-dimension approach for chi-squared divergence, enabling efficient robust policy learning. The authors provide instance-dependent and instance-independent suboptimality bounds and prove an information-theoretic lower bound, demonstrating near-optimality, complemented by experiments on simulated linear MDPs and an American Put Option to validate robustness and computational efficiency. This work advances robust offline RL by replacing traditional uncertainty sets with linearly structured regularization, offering practical benefits in high-dimensional settings while preserving strong theoretical guarantees.

Abstract

The Robust Regularized Markov Decision Process (RRMDP) is proposed to learn policies robust to dynamics shifts by adding regularization to the transition dynamics in the value function. Existing methods mostly use unstructured regularization, potentially leading to conservative policies under unrealistic transitions. To address this limitation, we propose a novel framework, the -rectangular linear RRMDP (-RRMDP), which introduces latent structures into both transition kernels and regularization. We focus on offline reinforcement learning, where an agent learns policies from a precollected dataset in the nominal environment. We develop the Robust Regularized Pessimistic Value Iteration (R2PVI) algorithm that employs linear function approximation for robust policy learning in -RRMDPs with -divergence based regularization terms on transition kernels. We provide instance-dependent upper bounds on the suboptimality gap of R2PVI policies, demonstrating that these bounds are influenced by how well the dataset covers state-action spaces visited by the optimal robust policy under robustly admissible transitions. We establish information-theoretic lower bounds to verify that our algorithm is near-optimal. Finally, numerical experiments validate that R2PVI learns robust policies and exhibits superior computational efficiency compared to baseline methods.

Paper Structure

This paper contains 59 sections, 20 theorems, 181 equations, 3 figures, 2 tables, 5 algorithms.

Table of Contents

  1. Introduction
  2. Related Work
  3. Problem Formulation
  4. Markov decision process (MDP).
  5. Robust regularized MDP (RRMDP)
  6. The $d$-rectangular linear RRMDP ($d$-RRMDP).
  7. Dynamic programming principles for $d$-RRMDPs
  8. Offline dataset and learning goal.
  9. Robust Regularized Pessimistic Value Iteration (R2PVI)
  10. R2PVI with the TV-Divergence
  11. R2PVI with the KL-Divergence
  12. R2PVI with the $\chi^2$-Divergence
  13. Suboptimality Analysis
  14. Instance-Dependent Upper Bound
  15. Instance-Independent Upper Bound
  16. ...and 44 more sections

Key Result

Proposition 3.2

(Robust Regularized Bellman Equation) Under the $d$-rectangular linear RRMDP, for any policy $\pi$ and any $(h,s,a) \in [H]\times {\mathcal{S}} \times \mathcal{A}$, we have

Figures (3)

  • Figure 1: Simulated results for linear MDP. In \ref{['fig:linear mdp - comparison']} and \ref{['fig:linear mdp - lambda']}, the $x$-axis refers to the perturbation in the testing environment. In \ref{['fig:linear mdp - rho and lambda']}, the $x$-axis represents different robust level $\rho$ and regularized penalty $\lambda$, respectively. \ref{['fig: robustness']} shows the robustness of algorithms under different robust level $\rho$ (DRPVI) or regularization penalty $\lambda$ (R2PVI).
  • Figure 2: Simulation results for the simulated American put option task. \ref{['fig: excution time w.r.t N']} shows the computation time of R2PVI with respect to the sample size $N$. \ref{['fig:exution time w.r.t. d']} shows the computation time of algorithms with respect to the feature dimension $d$.
  • Figure 3: The nominal environment and the worst case environment. The value on each arrow represents the transition probability. The MDP has two states, $s_1$ and $s_2$, and $H$ steps. For he nominal environment on the left, the $s_1$ is the good state where the transition is determined by an error term $\epsilon$, and $s_2$ is a fail state with reward 0 and only transitions to itself. The worst case environment on the right is obtained by perturbing the transition probability at each step of the nominal environment. The magnitude of the perturbation $\Delta_h^\lambda(\epsilon,D)$ at each stage $h$ depends on the divergence metric $D$, the regularized $\lambda$ and the parameter $\epsilon$.

Theorems & Definitions (26)

  • Proposition 3.2
  • Proposition 3.3
  • Proposition 4.1
  • Remark 4.2
  • Proposition 4.3
  • Remark 4.4
  • Remark 4.5
  • Proposition 4.6
  • Remark 4.7
  • Proposition 4.8
  • ...and 16 more