Who is Helping Whom? Analyzing Inter-dependencies to Evaluate Cooperation in Human-AI Teaming

TL;DR

The paper tackles the gap that task rewards inadequately quantify cooperation in human-AI teams. It introduces constructive interdependence, a domain-agnostic metric that captures sequential and reciprocal action dependencies by mapping Markov-game trajectories to STRIPS-style representations. Through Overcooked experiments with zero-shot coordination agents and human partners, it shows that high task rewards often mask poor cooperation, especially in Non-RC settings, and that RC settings exhibit stronger alignment between reward and cooperation. It demonstrates that incorporating the teaming metric into training can enhance cooperative behavior without harming task performance, offering a practical path to richer human-AI collaboration. The work provides a domain-agnostic toolkit and empirical evidence that interdependence metrics are essential for evaluating and improving real-world human-agent teamwork.

Abstract

State-of-the-art methods for Human-AI Teaming and Zero-shot Cooperation focus on task completion, i.e., task rewards, as the sole evaluation metric while being agnostic to how the two agents work with each other. Furthermore, subjective user studies only offer limited insight into the quality of cooperation existing within the team. Specifically, we are interested in understanding the cooperative behaviors arising within the team when trained agents are paired with humans -- a problem that has been overlooked by the existing literature. To formally address this problem, we propose the concept of constructive interdependence -- measuring how much agents rely on each other's actions to achieve the shared goal -- as a key metric for evaluating cooperation in human-agent teams. We interpret interdependence in terms of action interactions in a STRIPS formalism, and define metrics that allow us to assess the degree of reliance between the agents' actions. We pair state-of-the-art agents HAT with learned human models as well as human participants in a user study for the popular Overcooked domain, and evaluate the task reward and teaming performance for these human-agent teams. Our results demonstrate that although trained agents attain high task rewards, they fail to induce cooperative behavior, showing very low levels of interdependence across teams. Furthermore, our analysis reveals that teaming performance is not necessarily correlated with task reward, highlighting that task reward alone cannot reliably measure cooperation arising in a team.

Figures (10)

• Figure 1: Depicted are two strategies to fill a pot with onions in a cooking game. The coordinated strategy (right) is more efficient than the individual strategies (left), but runs the risk of failure if cooperation is not achieved.
• Figure 2: Left: Forced coordination layout which is a required cooperation (RC) setting. Right: Counter circuit layout which is an non-required cooperation (Non-RC) setting.
• Figure 3: Training curves comparing agents trained with the original task-only reward (orange curves) and the modified reward (blue curves) that incorporates interdependence ($\alpha$ = 0.3). All agents are trained for 5 million timesteps. The orange curves correspond to training with $\textbf{r}_{original}$, while the blue curves correspond to training with $\textbf{r}_{modified}$. Across all three algorithms (FCP, MEP, HSP), adding the teaming reward leads to a consistent increase in construcive interdependencies, while task performance remains unchanged.
• Figure 4: Software architecture for our domain-agnostic cooperation analysis framework. The Mapping Module converts raw trajectories to symbolic STRIPS-style traces, and the Analysis Module identifies interdependencies
• Figure 5: Illustration of an instance of the Search and Rescue Domain

Theorems & Definitions (5)

• Definition 1
• Definition 2
• Definition 3
• Definition 4
• Definition 5