Sparse Offline Reinforcement Learning with Corruption Robustness
Nam Phuong Tran, Andi Nika, Goran Radanovic, Long Tran-Thanh, Debmalya Mandal
TL;DR
This work addresses corruption-robust offline reinforcement learning in high-dimensional sparse MDPs, where an adversary can contaminate a fraction of data. It shows that traditional LSVI struggles under weak coverage due to pointwise pessimism, and proposes sparsity-aware pessimistic actor-critic methods that integrate sparse robust regression oracles to achieve non-vacuous guarantees under single-policy concentrability, even with contamination. The paper provides substantial results: non-vacuous bounds for sparse offline RL in high dimension, a detailed analysis of SRLE estimators under uniform coverage and without, and both uniform-coverage and single-concentrability performance guarantees for sparse AC methods. The findings establish a clear separation between LSVI and AC approaches in sparse, corrupted offline settings and point to computational considerations and future directions for relaxing combinatorial constraints in the critic.
Abstract
We investigate robustness to strong data corruption in offline sparse reinforcement learning (RL). In our setting, an adversary may arbitrarily perturb a fraction of the collected trajectories from a high-dimensional but sparse Markov decision process, and our goal is to estimate a near optimal policy. The main challenge is that, in the high-dimensional regime where the number of samples $N$ is smaller than the feature dimension $d$, exploiting sparsity is essential for obtaining non-vacuous guarantees but has not been systematically studied in offline RL. We analyse the problem under uniform coverage and sparse single-concentrability assumptions. While Least Square Value Iteration (LSVI), a standard approach for robust offline RL, performs well under uniform coverage, we show that integrating sparsity into LSVI is unnatural, and its analysis may break down due to overly pessimistic bonuses. To overcome this, we propose actor-critic methods with sparse robust estimator oracles, which avoid the use of pointwise pessimistic bonuses and provide the first non-vacuous guarantees for sparse offline RL under single-policy concentrability coverage. Moreover, we extend our results to the contaminated setting and show that our algorithm remains robust under strong contamination. Our results provide the first non-vacuous guarantees in high-dimensional sparse MDPs with single-policy concentrability coverage and corruption, showing that learning a near-optimal policy remains possible in regimes where traditional robust offline RL techniques may fail.