Offline Multi-task Transfer RL with Representational Penalization
Avinandan Bose, Simon Shaolei Du, Maryam Fazel
TL;DR
This work tackles offline multi-task reinforcement learning with a shared representation across $K$ source tasks for a target task modeled as a $d$-dimensional low-rank MDP. It introduces a data-dependent, pointwise uncertainty measure via an effective occupancy density and proves a bound on the target-policy suboptimality, showing that diverse source-task exploration can compensate for incomplete coverage. The authors propose a joint representation learning approach via maximum likelihood estimation across source tasks and a pessimistic transfer planning algorithm, PRT, that penalizes uncertainty in the learned representation. Theoretical guarantees, including a data-dependent suboptimality bound and a corollary on sample complexity, are complemented by experiments on a rich-observation Comblock environment demonstrating improved performance when transfer is uncertainty-aware. The results advance safe, data-efficient offline transfer by relaxing uniform coverage assumptions and highlighting how uncertainty quantification guides effective policy learning for the target task.
Abstract
We study the problem of representation transfer in offline Reinforcement Learning (RL), where a learner has access to episodic data from a number of source tasks collected a priori, and aims to learn a shared representation to be used in finding a good policy for a target task. Unlike in online RL where the agent interacts with the environment while learning a policy, in the offline setting there cannot be such interactions in either the source tasks or the target task; thus multi-task offline RL can suffer from incomplete coverage. We propose an algorithm to compute pointwise uncertainty measures for the learnt representation, and establish a data-dependent upper bound for the suboptimality of the learnt policy for the target task. Our algorithm leverages the collective exploration done by source tasks to mitigate poor coverage at some points by a few tasks, thus overcoming the limitation of needing uniformly good coverage for a meaningful transfer by existing offline algorithms. We complement our theoretical results with empirical evaluation on a rich-observation MDP which requires many samples for complete coverage. Our findings illustrate the benefits of penalizing and quantifying the uncertainty in the learnt representation.