CLIP-RL: Aligning Language and Policy Representations for Task Transfer in Reinforcement Learning
Chainesh Gautam, Raghuram Bharadwaj Diddigi
TL;DR
The paper addresses transferring RL policies across tasks described by natural language. It introduces CLIP-RL, a cross-modal transfer method that learns a shared embedding space for language instructions and policy representations to capture task structure rather than superficial textual similarity. It compares a baseline language-similarity transfer with a CLIP-inspired approach that trains a multi-modal encoder on base task pairs and uses diagonal cross-task similarities to initialize target policies. Experiments on grid-world environments show faster transfer and improved scalability for the CLIP-based method, highlighting its potential for efficient multi-task instruction-driven RL.
Abstract
Recently, there has been an increasing need to develop agents capable of solving multiple tasks within the same environment, especially when these tasks are naturally associated with language. In this work, we propose a novel approach that leverages combinations of pre-trained (language, policy) pairs to establish an efficient transfer pipeline. Our algorithm is inspired by the principles of Contrastive Language-Image Pretraining (CLIP) in Computer Vision, which aligns representations across different modalities under the philosophy that ''two modalities representing the same concept should have similar representations.'' The central idea here is that the instruction and corresponding policy of a task represent the same concept, the task itself, in two different modalities. Therefore, by extending the idea of CLIP to RL, our method creates a unified representation space for natural language and policy embeddings. Experimental results demonstrate the utility of our algorithm in achieving faster transfer across tasks.