Implicitly Aligning Humans and Autonomous Agents through Shared Task Abstractions

TL;DR

This work addresses the challenge of zero-shot coordination between humans and autonomous agents by introducing HA$^2$, a Hierarchical Ad Hoc Agent framework built on hierarchical reinforcement learning. By separating high-level task synchronization (Manager) from low-level sub-task execution (Worker) and enforcing shared, human-aligned task abstractions, HA$^2$ achieves robust generalization across unseen partners and layout changes while improving coordination with real humans. Empirical results in Overcooked show HA$^2$ outperforms strong baselines and state-of-the-art methods, with statistically significant gains and favorable human judgments of fluency, trust, and cooperativeness. The findings highlight the practical value of human-interpretable task hierarchies for resilient, efficient, and understandable human-AI collaboration, and point to future work on explicit mental models and improved human-agent communication.

Abstract

In collaborative tasks, autonomous agents fall short of humans in their capability to quickly adapt to new and unfamiliar teammates. We posit that a limiting factor for zero-shot coordination is the lack of shared task abstractions, a mechanism humans rely on to implicitly align with teammates. To address this gap, we introduce HA$^2$: Hierarchical Ad Hoc Agents, a framework leveraging hierarchical reinforcement learning to mimic the structured approach humans use in collaboration. We evaluate HA$^2$ in the Overcooked environment, demonstrating statistically significant improvement over existing baselines when paired with both unseen agents and humans, providing better resilience to environmental shifts, and outperforming all state-of-the-art methods.

Figures (4)

• Figure 1: Depicted is a scenario in the Overcooked game where an agent is working with a new teammate. The agent could choose to play a coordinated strategy that is more efficient than an alternative individual strategy, but runs the risk of failure if cooperation is not achieved. Successful ad hoc teaming requires not only being able to perform multiple strategies, but also know when to apply the different strategies. Current SotA approaches subsume these decisions into a single black-box; in contrast, we propose that a structured approach to these decisions provides significant benefits.
• Figure 2: An overview of the HA$^2$ architecture. Similar to human behavior, an observation is initially processed by the Manager to decide on the next high-level sub-task. Subsequently, the Worker executes the necessary low-level actions to complete the sub-task.
• Figure 3: Average score of HA$^2$s and the BCP and FCP baselines when paired with humans on each of the layouts. Each round was $80$ seconds long at $5$ FPS (T=$400$ steps). Significance markers: *=p$<0.05$, **=p$<0.005$, ***=p$<0.0005$. The red line indicates the max human-human score achieved on that layout from oai normalized to 400 steps.
• Figure 4: Subset of results from the eight Likert-scale questions that participants answer after playing with each agent for the comparison between HA$^2$ and their baselines. Bars that are more blue indicate that people agree more strongly with the statement. Conversely, more red indicates that people disagreed more strongly with the statement. Significance markers: *=p$<0.05$, **=p$<0.005$, ***=p$<0.0005$. Legend: SD=Strongly Disagree, D=Disagree, WD=Weakly Disagree, N=Neutral, WA=Weakly Agree, A=Agree, SA=Strongly Agree.