Lifelong Reinforcement Learning with Similarity-Driven Weighting by Large Models
Zhiyi Huang, Xiaohan Shan, Jianmin Li
TL;DR
The paper tackles catastrophic forgetting in lifelong reinforcement learning by proposing Similarity-Driven Weighting (SDW), which uses offline-generated, large-language-model-derived functions to adapt training dynamics across a sequence of tasks. SDW introduces a Task Similarity Function and a Weight Calculation Function that map task descriptions to a multidimensional similarity vector and then to training parameters such as the replay buffer proportion and a strategy-consistency weight, enabling adaptive transfer while preserving prior knowledge. The framework decouples LLM usage from the training loop, reducing overhead, and demonstrates superior performance on MiniHack and Atari benchmarks compared with baseline lifelong RL methods, including stronger forgetting mitigation and transfer. These results suggest SDW's potential to scale to large task sequences and open-ended lifelong learning scenarios by leveraging LLMs for reasoning about task relationships and transferring knowledge more efficiently.
Abstract
Lifelong Reinforcement Learning (LRL) holds significant potential for addressing sequential tasks, but it still faces considerable challenges. A key difficulty lies in effectively preventing catastrophic forgetting and facilitating knowledge transfer while maintaining reliable decision-making performance across subsequent tasks in dynamic environments. To tackle this, we propose a novel framework, SDW (Similarity-Driven Weighting Framework), which leverages large-language-model-generated dynamic functions to precisely control the training process. The core of SDW lies in two functions pre-generated by large models: the task similarity function and the weight computation function. The task similarity function extracts multidimensional features from task descriptions to quantify the similarities and differences between tasks in terms of states, actions, and rewards. The weight computation function dynamically generates critical training parameters based on the similarity information, including the proportion of old task data stored in the Replay Buffer and the strategy consistency weight in the loss function, enabling an adaptive balance between learning new tasks and transferring knowledge from previous tasks. By generating function code offline prior to training, rather than relying on large-model inference during the training process, the SDW framework reduces computational overhead while maintaining efficiency in sequential task scenarios. Experimental results on Atari and MiniHack sequential tasks demonstrate that SDW significantly outperforms existing lifelong reinforcement learning methods.