Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Learning Generalizable Skills from Offline Multi-Task Data for Multi-Agent Cooperation

Sicong Liu, Yang Shu, Chenjuan Guo, Bin Yang

TL;DR

This work addresses generalization in offline multi-task MARL by introducing HiSSD, a hierarchical framework that separates learning into common skills for cooperative temporal knowledge and task-specific skills for per-task action execution. The high-level planner learns transferable skills via KL-divergence trajectory matching and a forward state predictor, while the low-level controller leverages $eta$-VAE-based task embeddings and contrastive learning to tailor actions per task, all trained offline with an implicit Q-learning objective. Empirical results on SMAC and MAMuJoCo demonstrate robust unseen-task performance and meaningful skill visualizations, validating improved policy transfer and adaptive execution. Overall, HiSSD advances offline multi-task MARL by enabling robust generalization and task-aware action control without online interaction.

Abstract

Learning cooperative multi-agent policy from offline multi-task data that can generalize to unseen tasks with varying numbers of agents and targets is an attractive problem in many scenarios. Although aggregating general behavior patterns among multiple tasks as skills to improve policy transfer is a promising approach, two primary challenges hinder the further advancement of skill learning in offline multi-task MARL. Firstly, extracting general cooperative behaviors from various action sequences as common skills lacks bringing cooperative temporal knowledge into them. Secondly, existing works only involve common skills and can not adaptively choose independent knowledge as task-specific skills in each task for fine-grained action execution. To tackle these challenges, we propose Hierarchical and Separate Skill Discovery (HiSSD), a novel approach for generalizable offline multi-task MARL through skill learning. HiSSD leverages a hierarchical framework that jointly learns common and task-specific skills. The common skills learn cooperative temporal knowledge and enable in-sample exploitation for offline multi-task MARL. The task-specific skills represent the priors of each task and achieve a task-guided fine-grained action execution. To verify the advancement of our method, we conduct experiments on multi-agent MuJoCo and SMAC benchmarks. After training the policy using HiSSD on offline multi-task data, the empirical results show that HiSSD assigns effective cooperative behaviors and obtains superior performance in unseen tasks.

Learning Generalizable Skills from Offline Multi-Task Data for Multi-Agent Cooperation

TL;DR

This work addresses generalization in offline multi-task MARL by introducing HiSSD, a hierarchical framework that separates learning into common skills for cooperative temporal knowledge and task-specific skills for per-task action execution. The high-level planner learns transferable skills via KL-divergence trajectory matching and a forward state predictor, while the low-level controller leverages -VAE-based task embeddings and contrastive learning to tailor actions per task, all trained offline with an implicit Q-learning objective. Empirical results on SMAC and MAMuJoCo demonstrate robust unseen-task performance and meaningful skill visualizations, validating improved policy transfer and adaptive execution. Overall, HiSSD advances offline multi-task MARL by enabling robust generalization and task-aware action control without online interaction.

Abstract

Learning cooperative multi-agent policy from offline multi-task data that can generalize to unseen tasks with varying numbers of agents and targets is an attractive problem in many scenarios. Although aggregating general behavior patterns among multiple tasks as skills to improve policy transfer is a promising approach, two primary challenges hinder the further advancement of skill learning in offline multi-task MARL. Firstly, extracting general cooperative behaviors from various action sequences as common skills lacks bringing cooperative temporal knowledge into them. Secondly, existing works only involve common skills and can not adaptively choose independent knowledge as task-specific skills in each task for fine-grained action execution. To tackle these challenges, we propose Hierarchical and Separate Skill Discovery (HiSSD), a novel approach for generalizable offline multi-task MARL through skill learning. HiSSD leverages a hierarchical framework that jointly learns common and task-specific skills. The common skills learn cooperative temporal knowledge and enable in-sample exploitation for offline multi-task MARL. The task-specific skills represent the priors of each task and achieve a task-guided fine-grained action execution. To verify the advancement of our method, we conduct experiments on multi-agent MuJoCo and SMAC benchmarks. After training the policy using HiSSD on offline multi-task data, the empirical results show that HiSSD assigns effective cooperative behaviors and obtains superior performance in unseen tasks.

Paper Structure

This paper contains 49 sections, 3 theorems, 22 equations, 4 figures, 18 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Preliminaries
  3. Cooperative Multi-Agent Reinforcement Learning
  4. Learning Generalizable Policy from Offline Multi-Task Data
  5. Method
  6. Offline Multi-Task MARL with Cooperative Skill Learning
  7. Learning High-Level Planner with Common Skills
  8. Learning Low-Level Controller with Task-Specific Skills
  9. Training and Evaluation
  10. Experiments
  11. Benchmarks and Datasets
  12. SMAC
  13. MAMuJoCo
  14. Baselines
  15. SMAC
  16. ...and 34 more sections

Key Result

Theorem 3.1

Denote a set of $N$ training tasks with their offline data $\mathcal{D}^\mathcal{T}$, $\mathcal{D}^i$ is the data of the sampled task. Let random variables $x$ be some observations sampled from $\mathcal{D}^i$, skill $z \sim g_\omega(z \vert x)$, $h(x,z)=\frac{g_\omega(z \vert x)}{p(z)}$, $p(\mathca where $x^j$ are observations sampled from task $\mathcal{D}^j$ and $i \in \{ 1, 2, \cdots , N \}$.

Figures (4)

  • Figure 1: Overall framework of Hi-SSD. HiSSD utilizes a hierarchical framework that jointly learns common and task-specific skills from offline multi-task data to improve multi-agent policy transfer. (a) The high-level planner with common skills. HiSSD integrates cooperative temporal knowledge into common skills and enables an offline exploration. (b) The low-level controller with task-specific skills. The task-specific knowledge can guide the action execution adaptively among tasks. HiSSD uses an implicit Q-learning objective to train the value network.
  • Figure 2: Visualization of learned common skills. We use HiSSD to collect trajectories on five tasks in SMAC and partition these trajectories into four time windows. Plots in each figure represent the distribution of chosen common skills. We use multiple time windows to indicate the task flow.
  • Figure 3: Visualization of learned task-specific skills. We use HiSSD to collect trajectories on five tasks in SMAC and partition these trajectories into four time windows. Plots represent the distribution of chosen task-specific skills. We use multiple time windows to indicate the task flow.
  • Figure 4: Overall framework of Hi-SSD in SMAC. HiSSD utilizes a hierarchical framework that jointly learns common and task-specific skills from offline multi-task data to improve multi-agent policy transfer. (a) The high-level planner with common skills. HiSSD integrates cooperative temporal knowledge into common skills and enables an offline exploration. (b) The low-level controller with task-specific skills. The task-specific knowledge can guide the action execution adaptively among tasks. (c) HiSSD uses an implicit Q-learning objective to train the value network.

Theorems & Definitions (3)

  • Theorem 3.1
  • Lemma A.1
  • Theorem A.1