Corruption-Robust Algorithms with Uncertainty Weighting for Nonlinear Contextual Bandits and Markov Decision Processes
Chenlu Ye, Wei Xiong, Quanquan Gu, Tong Zhang
TL;DR
This work advances corruption-robust reinforcement learning by extending uncertainty-weighted regression techniques from linear to nonlinear function approximation. It introduces CR-Eluder-UCB for nonlinear contextual bandits and CR-LSVI-UCB for episodic MDPs, both leveraging data-dependent weights to bound weighted uncertainty and maintain optimism in the presence of adversarial corruption. The authors establish additive regret in the corruption level $\zeta$, parameterized by eluder dimension and covering numbers of the function class, and they handle unknown $\zeta$ via adaptive tuning. A key contribution is a novel uncertainty estimator and a weight-control analysis that bounds the cumulative weighted uncertainty even for general function classes, enabling near-optimal or improved regret bounds across known and unknown corruption and in both bandit and MDP settings. The results provide a unified, computationally efficient framework for robust RL with general function approximation, with the potential to impact scalable, real-world systems facing adversarial perturbations.
Abstract
Despite the significant interest and progress in reinforcement learning (RL) problems with adversarial corruption, current works are either confined to the linear setting or lead to an undesired $\tilde{O}(\sqrt{T}ζ)$ regret bound, where $T$ is the number of rounds and $ζ$ is the total amount of corruption. In this paper, we consider the contextual bandit with general function approximation and propose a computationally efficient algorithm to achieve a regret of $\tilde{O}(\sqrt{T}+ζ)$. The proposed algorithm relies on the recently developed uncertainty-weighted least-squares regression from linear contextual bandit and a new weighted estimator of uncertainty for the general function class. In contrast to the existing analysis that heavily relies on the linear structure, we develop a novel technique to control the sum of weighted uncertainty, thus establishing the final regret bounds. We then generalize our algorithm to the episodic MDP setting and first achieve an additive dependence on the corruption level $ζ$ in the scenario of general function approximation. Notably, our algorithms achieve regret bounds either nearly match the performance lower bound or improve the existing methods for all the corruption levels and in both known and unknown $ζ$ cases.