Generalising Multi-Agent Cooperation through Task-Agnostic Communication

TL;DR

We address the inefficiency of task-specific communication strategies in cooperative MARL by proposing a task-agnostic communication framework that is environment-specific. The method pre-trains a permutation-invariant set autoencoder to map a variable number of agent observations into a fixed-size latent state s_hat from which policies are conditioned, enabling scalability and out-of-distribution detection. Theoretical contributions include convergence guarantees when the latent state perfectly captures observations and a bound on regret proportional to reconstruction error; empirically, the approach outperforms task-specific strategies on unseen tasks and scales to more agents. This work enables general-purpose robotic collaboration across tasks without retraining communication strategies and supports runtime anomaly detection, with practical implications for real-world multi-robot systems.

Abstract

Existing communication methods for multi-agent reinforcement learning (MARL) in cooperative multi-robot problems are almost exclusively task-specific, training new communication strategies for each unique task. We address this inefficiency by introducing a communication strategy applicable to any task within a given environment. We pre-train the communication strategy without task-specific reward guidance in a self-supervised manner using a set autoencoder. Our objective is to learn a fixed-size latent Markov state from a variable number of agent observations. Under mild assumptions, we prove that policies using our latent representations are guaranteed to converge, and upper bound the value error introduced by our Markov state approximation. Our method enables seamless adaptation to novel tasks without fine-tuning the communication strategy, gracefully supports scaling to more agents than present during training, and detects out-of-distribution events in an environment. Empirical results on diverse MARL scenarios validate the effectiveness of our approach, surpassing task-specific communication strategies in unseen tasks. Our implementation of this work is available at https://github.com/proroklab/task-agnostic-comms.

Figures (6)

• Figure 1: Learning and applying a task-agnostic communication strategy in MARL.Offline pre-training. We pre-train a set autoencoder with sets of observations collected from exploring an environment. Since there is no reward signal involved in sampling these observations, the autoencoder learns a task-agnostic representation. When a variable number of agent observations are encoded by the set encoder, the output is a fixed-size latent vector $\mathbf{\hat{s}}$ approximating the Markovian state $s$ in a Dec-MDP. Policy training. For each agent, we deploy the pretrained set encoder to encode the global observation (assembled via communication) into $\mathbf{\hat{s}}$ on the fly. We condition the behaviour policy on $\mathbf{\hat{s}}$.
• Figure 2: Method details. We collect global observations by exploring the environment in a task-agnostic manner. Using these observations, we pretrain a set autoencoder (with encoder $\phi$ and decoder $\phi^{-1}$) using a self-supervised reconstruction loss. Keeping the autoencoder weights frozen, we train policies $\pi^i$ on various tasks $\tau$. The input to $\pi$ is the approximation of the Markov state $\mathbf{\hat{s}}_t$ and the relevant agent's observation $o^i_t$.
• Figure 3: Tasks. Circuit (top), Discovery (middle), and Pursuit-Evasion (bottom).
• Figure 4: Task-Agnostic communication strategies lead to greater rewards in novel tasks. For each set of results, we report the mean and central 95% interval over 5 seeds. We trained for 6.4 million environment steps in Melting Pot tasks, and 12 million environment steps in VMAS tasks. We used the same number of steps to pre-train the task-specific strategy on a similar task.
• Figure 5: Our task-agnostic strategy scales out-of-distribution. We pre-trained the communication strategy with 1, 2, and 3 agents for 1M environment steps and trained the policy for 12M environment steps. For each set of results, we report mean and central 95% interval over 5 seeds.

Theorems & Definitions (4)

• theorem thmcountertheorem
• proof
• theorem thmcountertheorem
• proof