Towards Robust Model-Based Reinforcement Learning Against Adversarial Corruption
Chenlu Ye, Jiafan He, Quanquan Gu, Tong Zhang
TL;DR
This paper tackles adversarial corruption in model-based reinforcement learning by allowing an adversary to perturb transition dynamics and measuring the total corruption with $C$. It introduces corruption-robust online (CR-OMLE) and offline (CR-PMLE) algorithms that incorporate TV-based uncertainty weights via an information-ratio framework and TV-eluder dimension to achieve provable guarantees. The online method attains a regret of order $\tilde{\mathcal{O}}(\sqrt{T} + C)$ with a matching lower bound signaling optimal corruption dependence, while the offline method delivers instance-dependent and instance-independent suboptimality bounds under a uniform coverage condition, plus a matching lower bound in the corruption term. The work leverages maximum likelihood estimation with uncertainty weighting to handle corruption in the model and closes a gap for corruption-robust guarantees in model-based RL, with implications for robust planning under adversarial transitions.
Abstract
This study tackles the challenges of adversarial corruption in model-based reinforcement learning (RL), where the transition dynamics can be corrupted by an adversary. Existing studies on corruption-robust RL mostly focus on the setting of model-free RL, where robust least-square regression is often employed for value function estimation. However, these techniques cannot be directly applied to model-based RL. In this paper, we focus on model-based RL and take the maximum likelihood estimation (MLE) approach to learn transition model. Our work encompasses both online and offline settings. In the online setting, we introduce an algorithm called corruption-robust optimistic MLE (CR-OMLE), which leverages total-variation (TV)-based information ratios as uncertainty weights for MLE. We prove that CR-OMLE achieves a regret of $\tilde{\mathcal{O}}(\sqrt{T} + C)$, where $C$ denotes the cumulative corruption level after $T$ episodes. We also prove a lower bound to show that the additive dependence on $C$ is optimal. We extend our weighting technique to the offline setting, and propose an algorithm named corruption-robust pessimistic MLE (CR-PMLE). Under a uniform coverage condition, CR-PMLE exhibits suboptimality worsened by $\mathcal{O}(C/n)$, nearly matching the lower bound. To the best of our knowledge, this is the first work on corruption-robust model-based RL algorithms with provable guarantees.