A Model-Based Solution to the Offline Multi-Agent Reinforcement Learning Coordination Problem

TL;DR

The paper tackles offline coordination in multi-agent RL, identifying Strategy Agreement and Strategy Fine-Tuning as the two core hurdles when no new interactions are possible. It introduces MOMA-PPO, the first model-based offline MARL method, which learns a centralized world-model ensemble to synthesize inter-agent data and trains policies with MAPPO, incorporating uncertainty penalties and adaptive rollouts to prevent model exploitation. Across offline coordination benchmarks, including Iterated Coordination Game and offline MAMuJoCo tasks with partial observability, MOMA-PPO consistently outperforms model-free baselines, demonstrating robust coordination and policy fine-tuning. The work highlights the practical value of world-model-based collaboration for offline multi-agent systems and points to future work in refining world models and extending to broader domains.

Abstract

Training multiple agents to coordinate is an essential problem with applications in robotics, game theory, economics, and social sciences. However, most existing Multi-Agent Reinforcement Learning (MARL) methods are online and thus impractical for real-world applications in which collecting new interactions is costly or dangerous. While these algorithms should leverage offline data when available, doing so gives rise to what we call the offline coordination problem. Specifically, we identify and formalize the strategy agreement (SA) and the strategy fine-tuning (SFT) coordination challenges, two issues at which current offline MARL algorithms fail. Concretely, we reveal that the prevalent model-free methods are severely deficient and cannot handle coordination-intensive offline multi-agent tasks in either toy or MuJoCo domains. To address this setback, we emphasize the importance of inter-agent interactions and propose the very first model-based offline MARL method. Our resulting algorithm, Model-based Offline Multi-Agent Proximal Policy Optimization (MOMA-PPO) generates synthetic interaction data and enables agents to converge on a strategy while fine-tuning their policies accordingly. This simple model-based solution solves the coordination-intensive offline tasks, significantly outperforming the prevalent model-free methods even under severe partial observability and with learned world models.

Figures (10)

• Figure 1: Comparing online learning and offline learning in policy space to illustrate the offline coordination problem. (a) During online learning, agents continuously interact using their current policies and collect new data that informs the next update of the co-evolution from $(\pi_1^0, \pi_2^0$) to $(\pi_1^a, \pi_2^a$). (b) During offline learning, agents cannot collect new data and thus, they can only estimate updates from the dataset of interactions (here collected with $\pi_1^{{\mathcal{D}}}$ and $\pi_2^{{\mathcal{D}}}$). To reach an optimal strategy agents must (1) agree on which optimum to target between $\star^a$ or $\star^b$, -- i.e. solve strategy agreement --, and (2) respectively derive the policy corresponding to that strategy ($\pi_1^{\star j}$ and $\pi_2^{\star j}$), -- i.e. solve strategy fine-tuning.
• Figure 2: Model-based rollouts generation (blue) from dataset's states (grey). Red denotes early termination and $k=3$.
• Figure 3: Environments illustrations. (a) Pay off matrix of the Iterated Coordination Game. (b) Two-agent Reacher, red and blue agents respectively control the torque on $\theta_1$ and $\theta_2$. (c) Four-agent Ant, each agent controls a different limb (shown with different colors). In PO tasks, agents only observe the limb they control while the torso observations -- in white -- are available only to the yellow agent.
• Figure 4: Comparision between using epistemic uncertainty reward penalty (MOMA-PPO) vs. aleatoric uncertainty reward penalty (MOPO-like). Mean and standard error of the mean on three seeds.
• Figure 5: Learning Curves for two-agent Reacher. Mean and standard error of the mean on three seeds.