Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Tackling Cooperative Incompatibility for Zero-Shot Human-AI Coordination

Yang Li, Shao Zhang, Jichen Sun, Wenhao Zhang, Yali Du, Ying Wen, Xinbing Wang, Wei Pan

TL;DR

COLE tackles cooperative incompatibility in zero-shot human-AI coordination by recasting cooperative tasks as Graphic-Form Games and Preference Graphic-Form Games, enabling principled assessment of how well new strategies are preferred by others. It introduces two practical solvers, COLE_SV and COLE_R, and a trainer to iteratively generate best-preferred strategies within a cooperative-incompatibility mixture, with theoretical guarantees of convergence to a local best-preferred strategy at a $Q$-sublinear rate under in-degree centrality. The framework is instantiated in the Overcooked environment via an online COLE platform and validated through a human-AI study with 130 participants, showing clear subjective and objective advantages over state-of-the-art baselines. The work provides a scalable, open-ended learning pipeline for zero-shot human-AI coordination, along with an extensive ablation and human-evaluation program. Collectively, COLE offers a practical, theoretically grounded path to robust coordination with unseen human and AI partners in complex cooperative settings.

Abstract

Securing coordination between AI agent and teammates (human players or AI agents) in contexts involving unfamiliar humans continues to pose a significant challenge in Zero-Shot Coordination. The issue of cooperative incompatibility becomes particularly prominent when an AI agent is unsuccessful in synchronizing with certain previously unknown partners. Traditional algorithms have aimed to collaborate with partners by optimizing fixed objectives within a population, fostering diversity in strategies and behaviors. However, these techniques may lead to learning loss and an inability to cooperate with specific strategies within the population, a phenomenon named cooperative incompatibility in learning. In order to solve cooperative incompatibility in learning and effectively address the problem in the context of ZSC, we introduce the Cooperative Open-ended LEarning (COLE) framework, which formulates open-ended objectives in cooperative games with two players using perspectives of graph theory to evaluate and pinpoint the cooperative capacity of each strategy. We present two practical algorithms, specifically \algo and \algoR, which incorporate insights from game theory and graph theory. We also show that COLE could effectively overcome the cooperative incompatibility from theoretical and empirical analysis. Subsequently, we created an online Overcooked human-AI experiment platform, the COLE platform, which enables easy customization of questionnaires, model weights, and other aspects. Utilizing the COLE platform, we enlist 130 participants for human experiments. Our findings reveal a preference for our approach over state-of-the-art methods using a variety of subjective metrics. Moreover, objective experimental outcomes in the Overcooked game environment indicate that our method surpasses existing ones when coordinating with previously unencountered AI agents and the human proxy model.

Tackling Cooperative Incompatibility for Zero-Shot Human-AI Coordination

TL;DR

COLE tackles cooperative incompatibility in zero-shot human-AI coordination by recasting cooperative tasks as Graphic-Form Games and Preference Graphic-Form Games, enabling principled assessment of how well new strategies are preferred by others. It introduces two practical solvers, COLE_SV and COLE_R, and a trainer to iteratively generate best-preferred strategies within a cooperative-incompatibility mixture, with theoretical guarantees of convergence to a local best-preferred strategy at a -sublinear rate under in-degree centrality. The framework is instantiated in the Overcooked environment via an online COLE platform and validated through a human-AI study with 130 participants, showing clear subjective and objective advantages over state-of-the-art baselines. The work provides a scalable, open-ended learning pipeline for zero-shot human-AI coordination, along with an extensive ablation and human-evaluation program. Collectively, COLE offers a practical, theoretically grounded path to robust coordination with unseen human and AI partners in complex cooperative settings.

Abstract

Securing coordination between AI agent and teammates (human players or AI agents) in contexts involving unfamiliar humans continues to pose a significant challenge in Zero-Shot Coordination. The issue of cooperative incompatibility becomes particularly prominent when an AI agent is unsuccessful in synchronizing with certain previously unknown partners. Traditional algorithms have aimed to collaborate with partners by optimizing fixed objectives within a population, fostering diversity in strategies and behaviors. However, these techniques may lead to learning loss and an inability to cooperate with specific strategies within the population, a phenomenon named cooperative incompatibility in learning. In order to solve cooperative incompatibility in learning and effectively address the problem in the context of ZSC, we introduce the Cooperative Open-ended LEarning (COLE) framework, which formulates open-ended objectives in cooperative games with two players using perspectives of graph theory to evaluate and pinpoint the cooperative capacity of each strategy. We present two practical algorithms, specifically \algo and \algoR, which incorporate insights from game theory and graph theory. We also show that COLE could effectively overcome the cooperative incompatibility from theoretical and empirical analysis. Subsequently, we created an online Overcooked human-AI experiment platform, the COLE platform, which enables easy customization of questionnaires, model weights, and other aspects. Utilizing the COLE platform, we enlist 130 participants for human experiments. Our findings reveal a preference for our approach over state-of-the-art methods using a variety of subjective metrics. Moreover, objective experimental outcomes in the Overcooked game environment indicate that our method surpasses existing ones when coordinating with previously unencountered AI agents and the human proxy model.
Paper Structure (51 sections, 5 theorems, 14 equations, 19 figures, 4 tables, 2 algorithms)

This paper contains 51 sections, 5 theorems, 14 equations, 19 figures, 4 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Related Works
  3. Zero-Shot Human-AI Coordination.
  4. Open-Ended Learning.
  5. Coordination Graphs.
  6. Preliminaries
  7. Normal-Form Game.
  8. Open-Ended Meta-Games and Analysis.
  9. Cooperative Theoretic Concepts.
  10. Graph Theoretic Concepts.
  11. Cooperative Open-Ended Learning
  12. Graphic-Form Games (GFGs)
  13. Motivating Examples
  14. Cooperative Incompatibility
  15. Cooperative Incompatibility in the learning of ZSC algorithms.
  16. ...and 36 more sections

Key Result

Theorem 4.4

Let $s_0\in {\mathcal{S}}$ be the initial strategy and $s_i=\operatorname{oracle}(s_{i-1})$ for $i \in \mathbb{N}$. Under the effective functioning of the approximated oracle as characterized by Eq. eq:oracle_approx, we can say that the sequence $\{s_i\}$ for ${i\in \mathbb{N}}$ could converge to a

Figures (19)

  • Figure 1: The Game Graph, (sub-) preference graph and corresponding preference centrality matrix. The (sub-) preference graphs are for all four iterations in the training process, and the corresponding preference in-degree centrality matrix is based on them. As can be observed in ${\mathcal{G}}^\prime_3$ and ${\mathcal{G}}^\prime_4$ in (a), the newly updated strategies fail to be preferred by others and have centrality values of 1, despite an increase in the mean of rewards with all others. In (b), we illustrate an ideal learning process in which a newly generated strategy can achieve higher outcomes than all previous strategies.
  • Figure 2: Motivating examples. Fig. \ref{['fig:hsp_eg']} features an analysis conducted by HSP HSP. The FCP agent converges to a fixed pattern of exclusively preparing onion soup, thereby failing to establish coordination with a human participant who prefers making tomato soup. Fig. \ref{['fig:mep_eta']} shows a training process of the MEP algorithm with several comparative incompatibility problems. The payoff matrix of each strategy during training and the corresponding preference centrality matrix of the MEP algorithm in the Overcooked. A deeper shade of red in the payoff matrix signifies higher utility. The darker the color in the preference centrality matrix, the lower the centrality value, and the more other strategies prefer it.
  • Figure 3: An overview of one generation in COLE framework: The solver derives the cooperative incompatible distribution $\phi$ using a cooperative incompatibility solver, which can be any algorithm that evaluates cooperative contribution. The trainer then approximates the relaxed best response by optimizing individual and cooperative compatible objectives. The oracle's training data is generated using partners selected based on the cooperative incompatibility distribution and the agent's strategy. Finally, the approximated strategy $s_{n+1}$ is added to the population, and the next generation begins.
  • Figure 4: The illustrated figure characterizes the conceptual architecture of the proposed human-AI experimental pipeline, structured around six critical stages. (Stage 1) Experiment Statement delineates the nature of the experiment, associated risks, and ethical considerations among other relevant aspects. (Stage 2) Before-game Survey principally focuses on the acquisition of participant information. Delving deeper, (Stage 3) Experimental Instructions dispenses extensive procedural guidelines for the experiment. Subsequently, participants are invited to engage in a sequence of trial games to acquaint themselves with the experimental procedure in (Stage 4) Game Tutorial. Proceeding to (Stage 5) Main Experiment, it entails various rounds with a diverse array of differentiated AI agents. Post each round, participants are obliged to fill out a survey. The entire experimental process culminates with a comprehensive evaluation of the collaborative AI counterparts in (Stage 6) Final Survey. The pipeline is integrated into one platform designed for seamless interaction. Researchers have the flexibility to tailor the pipeline according to their needs, while participants benefit from a user-friendly interface that enables them to complete the stages with ease.
  • Figure 5: Overcooked environment layouts. The Cramped Rm., Asymm. Adv., and Coord. Ring layouts are more conducive to higher rewards when players cooperate with different partners. On the other hand, the Forced Coord., Counter Circ., and Asymm. Adv. layouts offer distinct designs that serve as ideal testbeds to explore and foster cooperation between players.
  • ...and 14 more figures

Theorems & Definitions (13)

  • Definition 4.1: Graphic-Form Game
  • Definition 4.2: Preference Centrality
  • Definition 4.3: Cooperative Incompatibility
  • Theorem 4.4
  • proof
  • Corollary 4.4
  • proof
  • Theorem A.1
  • proof
  • Lemma A.1
  • ...and 3 more