Hybrid Training for Enhanced Multi-task Generalization in Multi-agent Reinforcement Learning

TL;DR

HyGen tackles the challenge of multi-task generalization in cooperative MARL by coupling offline general skill discovery with online refinement under CTDE. It first extracts general skills from multi-task offline data via a global trajectory encoder with multi-head attention and an action decoder, framing skills as $z_i \in \mathcal{Z}$, then trains a high-level policy with a linearly decaying offline-online ratio $R_h$ in a QMIX-style framework to sequence and refine these skills. The approach also includes a local-consistent skill encoder and a dynamic CQL loss to address out-of-distribution concerns, enabling effective zero-shot execution on unseen SMAC tasks. Empirical results on SMAC show HyGen outperforms both online and offline baselines, with strong generalization to unseen tasks and improved sample efficiency, highlighting the practical potential of hybrid offline-online skill-based MARL. Future work may explore integrating large language models to further boost cross-task adaptability and scalability.

Abstract

In multi-agent reinforcement learning (MARL), achieving multi-task generalization to diverse agents and objectives presents significant challenges. Existing online MARL algorithms primarily focus on single-task performance, but their lack of multi-task generalization capabilities typically results in substantial computational waste and limited real-life applicability. Meanwhile, existing offline multi-task MARL approaches are heavily dependent on data quality, often resulting in poor performance on unseen tasks. In this paper, we introduce HyGen, a novel hybrid MARL framework, Hybrid Training for Enhanced Multi-Task Generalization, which integrates online and offline learning to ensure both multi-task generalization and training efficiency. Specifically, our framework extracts potential general skills from offline multi-task datasets. We then train policies to select the optimal skills under the centralized training and decentralized execution paradigm (CTDE). During this stage, we utilize a replay buffer that integrates both offline data and online interactions. We empirically demonstrate that our framework effectively extracts and refines general skills, yielding impressive generalization to unseen tasks. Comparative analyses on the StarCraft multi-agent challenge show that HyGen outperforms a wide range of existing solely online and offline methods.

Figures (7)

• Figure 1: The overall framework of HyGen is structured as follows: (1) Initially, HyGen learns a global trajectory encoder and action decoders from multi-task data to discover general skills applicable across different tasks. (2) HyGen then learns high-level policies utilizing a hybrid replay buffer that incorporates both offline data and online interactions, essentially refining the skills discovered in the initial stage. (3) During zero-shot execution, HyGen selects and sequences these skills based on a high-level policy and decodes specific actions through the action decoder.
• Figure 2: Training framework during the general skill discovery phase of HyGen. The global trajectory encoder extracts a set of general skills common across different tasks from multi-task offline datasets, while the action decoders learn to delineate different agent actions within the discovered skills. The global trajectory encoder uses a task decomposer and multi-head self-attention to handle varying input from different tasks.
• Figure 3: Training framework during the high-level policy learning phase of HyGen. The hybrid buffer contains trajectories from the online buffer $\mathcal{R}$ and the offline dataset $\mathcal{D}$. The observation encoder extracts representations from local information. Meanwhile, the mixing network employs self-attention to accommodate varying input dimensions across different tasks.
• Figure 4: Comparison of HyGen, QMIX, and ODIS on the source task $3m$.
• Figure 5: Average test win rates of HyGen using a linearly decreasing hybrid ratio and three fixed hybrid ratios—20%, 50%, and 80%—in the marine-hard task set with the medium dataset. All experiment results were conducted over five random seeds.