Cross-environment Cooperation Enables Zero-shot Multi-agent Coordination

TL;DR

This work addresses zero-shot multi-agent coordination by training agents across a broad distribution of environments rather than relying on partner diversity. It introduces Cross-Environment Cooperation (CEC) and two Jax-based procedural generators to create billions of solvable coordination challenges, enabling self-play across many tasks. Empirical results in toy and complex domains (including Overcooked) and human studies show that environment diversity yields stronger cross-partner generalization than traditional population-based training, enabling zero-shot coordination with novel partners. The findings suggest a practical path toward generalist cooperative agents without human data, while acknowledging trade-offs in training efficiency and potential limits in absolute performance.

Abstract

Zero-shot coordination (ZSC), the ability to adapt to a new partner in a cooperative task, is a critical component of human-compatible AI. While prior work has focused on training agents to cooperate on a single task, these specialized models do not generalize to new tasks, even if they are highly similar. Here, we study how reinforcement learning on a distribution of environments with a single partner enables learning general cooperative skills that support ZSC with many new partners on many new problems. We introduce two Jax-based, procedural generators that create billions of solvable coordination challenges. We develop a new paradigm called Cross-Environment Cooperation (CEC), and show that it outperforms competitive baselines quantitatively and qualitatively when collaborating with real people. Our findings suggest that learning to collaborate across many unique scenarios encourages agents to develop general norms, which prove effective for collaboration with different partners. Together, our results suggest a new route toward designing generalist cooperative agents capable of interacting with humans without requiring human data.

Figures (31)

• Figure 1: Overview of learning general coordination through Cross-environment Cooperation (CEC). By training agents in self-play on a large distribution of environments, we find that agents develop the ability to coordinate with novel partners and novel problems, contrasting with prior work which suggests self-play is insufficient for learning general norms for cooperation.
• Figure 2: The Dual Destination Problem. In the fixed task (a), players start in opposite squares and must enter different green squares from each other to receive a reward. In the procedurally generated variation (b), the initial positions of the green goal cells and agents are randomized.
• Figure 3: Evaluation of IPPO and FCP baselines on the Fixed and Procedurally generated versions of the Dual Destination problem (error bars show the standard error of the mean). CEC generalizes better in both cases ($p<0.001$ for t-tests comparing CEC to both FCP and IPPO).
• Figure 4: Five original Overcooked layouts. Left to right: Asymmetric Advantages, Coordination Ring, Counter Circuit, Cramped Room, Forced Coordination.
• Figure 5: Sample from the billions of solvable, diverse Overcooked tasks created by our procedural environment generator.