Provable Zero-Shot Generalization in Offline Reinforcement Learning
Zhiyong Wang, Chen Yang, John C. S. Lui, Dongruo Zhou
TL;DR
This work addresses zero-shot generalization in offline reinforcement learning across multiple environments, showing that vanilla offline RL cannot generalize without environment context. It introduces pessimistic empirical risk minimization (PERM) and pessimistic proximal policy optimization (PPPO), both leveraging a pessimistic policy evaluation oracle to enable ZSG. The authors prove provable guarantees, decomposing the ZSG gap into a supervised-learning term and a reinforcement-learning term that depends on dataset coverage and uncertainty quantification, and provide specialization to offline linear MDPs with explicit bounds. The results establish a foundational understanding of generalization in offline RL and propose practical, theoretically sound algorithms for training policies that perform well on unseen environments. This work thus advances robust, context-aware offline RL with potential broad impact on multi-environment deployment of learned policies.
Abstract
In this work, we study offline reinforcement learning (RL) with zero-shot generalization property (ZSG), where the agent has access to an offline dataset including experiences from different environments, and the goal of the agent is to train a policy over the training environments which performs well on test environments without further interaction. Existing work showed that classical offline RL fails to generalize to new, unseen environments. We propose pessimistic empirical risk minimization (PERM) and pessimistic proximal policy optimization (PPPO), which leverage pessimistic policy evaluation to guide policy learning and enhance generalization. We show that both PERM and PPPO are capable of finding a near-optimal policy with ZSG. Our result serves as a first step in understanding the foundation of the generalization phenomenon in offline reinforcement learning.