Pessimism Principle Can Be Effective: Towards a Framework for Zero-Shot Transfer Reinforcement Learning
Chi Zhang, Ziying Jia, George K. Atia, Sihong He, Yue Wang
TL;DR
This work tackles zero-shot, multi-domain transfer reinforcement learning by introducing a pessimism principle that constructs conservative proxies for target-domain performance, guaranteeing a safe lower bound and reducing the risk of negative transfer. It develops two distributed, convergence-guaranteed proxies based on robust RL concepts, notably an Averaged Operator Based Proxy (AO) whose fixed point yields a conservative value function, and a Minimal Pessimism approach that selectively leverages the most informative source domains. The proposed MDTL-Avg and MDTL-Max algorithms enable scalable, privacy-preserving learning across multiple sources, with provable convergence rates and partial linear speedup in the number of domains. Experimental results across robotics, scheduling, and control tasks demonstrate improved target-domain performance, robustness to model uncertainty, and effective mitigation of negative transfer. Overall, the framework provides a principled, theoretically grounded path toward safe and reliable zero-shot TL in RL with practical distributed implementations.
Abstract
Transfer reinforcement learning aims to derive a near-optimal policy for a target environment with limited data by leveraging abundant data from related source domains. However, it faces two key challenges: the lack of performance guarantees for the transferred policy, which can lead to undesired actions, and the risk of negative transfer when multiple source domains are involved. We propose a novel framework based on the pessimism principle, which constructs and optimizes a conservative estimation of the target domain's performance. Our framework effectively addresses the two challenges by providing an optimized lower bound on target performance, ensuring safe and reliable decisions, and by exhibiting monotonic improvement with respect to the quality of the source domains, thereby avoiding negative transfer. We construct two types of conservative estimations, rigorously characterize their effectiveness, and develop efficient distributed algorithms with convergence guarantees. Our framework provides a theoretically sound and practically robust solution for transfer learning in reinforcement learning.