Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Corruption-Robust Offline Reinforcement Learning with General Function Approximation

Chenlu Ye, Rui Yang, Quanquan Gu, Tong Zhang

TL;DR

We address corruption-robust offline reinforcement learning with general function approximation by introducing a cumulative corruption level $\zeta$ and developing an uncertainty-weighted offline algorithm CR-PEVI. The method leverages an uncertainty-weight iteration to compute sample weights and constructs a pessimistic value-iteration scheme with a confidence set to control corruption effects. Theoretical results show a suboptimality bound with a corruption term of $\tilde{\mathcal{O}}(\zeta/(nC(\mathcal{F},\mu)))$, and in linear MDPs this term tightens to $\mathcal{O}(\zeta d/n)$, matching a lower bound. Empirically, a practical instantiation UWMSG improves performance under reward and dynamics corruption, highlighting robustness gains for offline RL in real-world settings.

Abstract

We investigate the problem of corruption robustness in offline reinforcement learning (RL) with general function approximation, where an adversary can corrupt each sample in the offline dataset, and the corruption level $ζ\geq0$ quantifies the cumulative corruption amount over $n$ episodes and $H$ steps. Our goal is to find a policy that is robust to such corruption and minimizes the suboptimality gap with respect to the optimal policy for the uncorrupted Markov decision processes (MDPs). Drawing inspiration from the uncertainty-weighting technique from the robust online RL setting \citep{he2022nearly,ye2022corruptionrobust}, we design a new uncertainty weight iteration procedure to efficiently compute on batched samples and propose a corruption-robust algorithm for offline RL. Notably, under the assumption of single policy coverage and the knowledge of $ζ$, our proposed algorithm achieves a suboptimality bound that is worsened by an additive factor of $\mathcal{O}(ζ(C(\widehat{\mathcal{F}},μ)n)^{-1})$ due to the corruption. Here $\widehat{\mathcal{F}}$ is the confidence set, and the dataset $\mathcal{Z}_n^H$, and $C(\widehat{\mathcal{F}},μ)$ is a coefficient that depends on $\widehat{\mathcal{F}}$ and the underlying data distribution $μ$. When specialized to linear MDPs, the corruption-dependent error term reduces to $\mathcal{O}(ζd n^{-1})$ with $d$ being the dimension of the feature map, which matches the existing lower bound for corrupted linear MDPs. This suggests that our analysis is tight in terms of the corruption-dependent term.

Corruption-Robust Offline Reinforcement Learning with General Function Approximation

TL;DR

We address corruption-robust offline reinforcement learning with general function approximation by introducing a cumulative corruption level and developing an uncertainty-weighted offline algorithm CR-PEVI. The method leverages an uncertainty-weight iteration to compute sample weights and constructs a pessimistic value-iteration scheme with a confidence set to control corruption effects. Theoretical results show a suboptimality bound with a corruption term of , and in linear MDPs this term tightens to , matching a lower bound. Empirically, a practical instantiation UWMSG improves performance under reward and dynamics corruption, highlighting robustness gains for offline RL in real-world settings.

Abstract

We investigate the problem of corruption robustness in offline reinforcement learning (RL) with general function approximation, where an adversary can corrupt each sample in the offline dataset, and the corruption level quantifies the cumulative corruption amount over episodes and steps. Our goal is to find a policy that is robust to such corruption and minimizes the suboptimality gap with respect to the optimal policy for the uncorrupted Markov decision processes (MDPs). Drawing inspiration from the uncertainty-weighting technique from the robust online RL setting \citep{he2022nearly,ye2022corruptionrobust}, we design a new uncertainty weight iteration procedure to efficiently compute on batched samples and propose a corruption-robust algorithm for offline RL. Notably, under the assumption of single policy coverage and the knowledge of , our proposed algorithm achieves a suboptimality bound that is worsened by an additive factor of due to the corruption. Here is the confidence set, and the dataset , and is a coefficient that depends on and the underlying data distribution . When specialized to linear MDPs, the corruption-dependent error term reduces to with being the dimension of the feature map, which matches the existing lower bound for corrupted linear MDPs. This suggests that our analysis is tight in terms of the corruption-dependent term.
Paper Structure (41 sections, 18 theorems, 146 equations, 5 figures, 3 tables, 3 algorithms)

This paper contains 41 sections, 18 theorems, 146 equations, 5 figures, 3 tables, 3 algorithms.

Table of Contents

  1. Introduction
  2. Related Work
  3. Preliminaries
  4. Episodic MDPs
  5. General Function Approximation
  6. Offline Data Collection Process
  7. Algorithm
  8. Uncertainty-Related Weights
  9. Corruption-Robust Algorithm
  10. Theoretical Analysis
  11. Coverage Condition
  12. Main Result
  13. Experiments
  14. Conclusion
  15. Proof of Theorem \ref{['th:mdp']}
  16. ...and 26 more sections

Key Result

Lemma 3.1

There exists a $T$ such that the output of Algorithm alg:wi$\{\sigma_i:=\sigma_i^{T+1}\}_{i=1}^n$ satisfy: where $\psi(z_i)=\sup_{f,f'\in{\mathcal{F}}}\frac{|f(z_i) - f'(z_i)|/\alpha}{\sqrt{\lambda + \sum_{j=1}^n(f(z_j) - f'(z_j))^2/\sigma_j^2}}$.

Figures (5)

  • Figure 1: Performance on the Walker2d and the Halfcheetah tasks under (a) random reward, (b) random dynamics, (c) adversarial reward, and (d) adversarial dynamics attacks.
  • Figure 2: Comparison on the halfcheetah task under (a) random reward, (b) random dynamics, (c) adversarial reward, and (d) adversarial dynamics attacks.
  • Figure 3: Comparison on the walker2d task under (a) random reward, (b) random dynamics, (c) adversarial reward, and (d) adversarial dynamics attacks.
  • Figure 4: Comparison on the hopper task under (a) random reward, (b) random dynamics, (c) adversarial reward, and (d) adversarial dynamics attacks.
  • Figure 5: Performance of UWMSG under varying levels of corruption. Results are averaged over 5 random seeds.

Theorems & Definitions (38)

  • Definition 2.1: $\epsilon$-Covering Number
  • Definition 2.2: Cumulative Corruption
  • Lemma 3.1
  • Definition 4.1: Coverage Coefficient
  • Theorem 1
  • Remark 4.1
  • Lemma 4.1
  • Theorem 2: Minimax Lower Bound for Linear MDPs
  • Lemma A.1: Regret Decomposition
  • proof
  • ...and 28 more