ORVIT: Near-Optimal Online Distributionally Robust Reinforcement Learning
Debamita Ghosh, George K. Atia, Yue Wang
TL;DR
This paper addresses the challenge of online distributionally robust reinforcement learning when deployment may differ from training due to environment shifts. It introduces RVI-$f$, a model-based, optimistic robust planning framework that operates without generative models or offline data and supports general $f$-divergence ambiguity sets, including $\chi^2$ and KL. The authors establish sublinear regret and near-optimal sample complexity for the online robust objective, and provide minimax lower bounds to demonstrate near-optimality under these uncertainty sets. They also validate the approach experimentally on Gambler’s and Frozen Lake tasks under distributional shifts, showing improved worst-case performance and practical robustness. Overall, the work advances online DRRL by removing strong coverage assumptions and delivering data-efficient, theoretically-grounded algorithms with solid empirical evidence for real-world robustness.
Abstract
We investigate reinforcement learning (RL) in the presence of distributional mismatch between training and deployment, where policies trained in simulators often underperform in practice due to mismatches between training and deployment conditions, and thereby reliable guarantees on real-world performance are essential. Distributionally robust RL addresses this issue by optimizing worst-case performance over an uncertainty set of environments and providing an optimized lower bound on deployment performance. However, existing studies typically assume access to either a generative model or offline datasets with broad coverage of the deployment environment-assumptions that limit their practicality in unknown environments without prior knowledge. In this work, we study a more practical and challenging setting: online distributionally robust RL, where the agent interacts only with a single unknown training environment while seeking policies that are robust with respect to an uncertainty set around this nominal model. We consider general $f$-divergence-based ambiguity sets, including $χ^2$ and KL divergence balls, and design a computationally efficient algorithm that achieves sublinear regret for the robust control objective under minimal assumptions, without requiring generative or offline data access. Moreover, we establish a corresponding minimax lower bound on the regret of any online algorithm, demonstrating the near-optimality of our method. Experiments across diverse environments with model misspecification show that our approach consistently improves worst-case performance and aligns with the theoretical guarantees.