Offline-to-Online Multi-Agent Reinforcement Learning with Offline Value Function Memory and Sequential Exploration

TL;DR

A novel O2O MARL framework called Offline Value Function Memory with Sequential Exploration (OVMSE) and a decentralized Sequential Exploration (SE) strategy tailored for O2O MARL, which effectively utilizes the pre-trained offline policy for exploration, thereby significantly reducing the joint state-action space to be explored.

Abstract

Offline-to-Online Reinforcement Learning has emerged as a powerful paradigm, leveraging offline data for initialization and online fine-tuning to enhance both sample efficiency and performance. However, most existing research has focused on single-agent settings, with limited exploration of the multi-agent extension, i.e., Offline-to-Online Multi-Agent Reinforcement Learning (O2O MARL). In O2O MARL, two critical challenges become more prominent as the number of agents increases: (i) the risk of unlearning pre-trained Q-values due to distributional shifts during the transition from offline-to-online phases, and (ii) the difficulty of efficient exploration in the large joint state-action space. To tackle these challenges, we propose a novel O2O MARL framework called Offline Value Function Memory with Sequential Exploration (OVMSE). First, we introduce the Offline Value Function Memory (OVM) mechanism to compute target Q-values, preserving knowledge gained during offline training, ensuring smoother transitions, and enabling efficient fine-tuning. Second, we propose a decentralized Sequential Exploration (SE) strategy tailored for O2O MARL, which effectively utilizes the pre-trained offline policy for exploration, thereby significantly reducing the joint state-action space to be explored. Extensive experiments on the StarCraft Multi-Agent Challenge (SMAC) demonstrate that OVMSE significantly outperforms existing baselines, achieving superior sample efficiency and overall performance.

Figures (4)

• Figure 1: Q-value evolution for different algorithms during the online phase, starting from pre-trained Q-value from offline learning. OVMSE clearly preserves the offline Q-value, while other methods experience rapid unlearning (i.e., online value estimations for these samples decrease rapidly).
• Figure 2: Median test win rates with respect to environment steps for OVMSE and baselines across different tasks pre-trained with medium or medium replay datasets. Our results demonstrate that OVMSE preserves offline knowledge, enables efficient exploration, and leads to faster fine-tuning and superior overall performance.
• Figure 3: Mean test returns for all tasks under different algorithms.
• Figure 4: We present the results of the ablation study and conclude that both OVM and SE are useful and effective.