On the Utility of External Agent Intention Predictor for Human-AI Coordination

TL;DR

This work tackles the difficulty of coordinating with opaque AI partners by introducing an external Theory of Mind (ToM) predictor that is trained offline from agent trajectories to forecast a target agent's future actions as a fixed-length sequence $l$ given history $H$ and current state $s_n$, and then visualizes these predictions in real time. The ToM predictor is implemented as a transformer-based model trained with self-play and cross-play data, using a history length $l_h=10$ and a prediction length $l=3$, and serves as a general plug-in that does not alter the target agent's behavior. Experiments in the Overcooked environment with SP and FCP agents show that human-AI teams achieve higher rewards and improved situational awareness when aided by the ToM predictor, though performance gains depend on layout and human variability. The results support the utility of external intention prediction for practical human-AI collaboration and highlight considerations for trust, prediction accuracy, and generalization to more complex domains, suggesting future work on semantics and safety.

Abstract

Reaching a consensus on the team plans is vital to human-AI coordination. Although previous studies provide approaches through communications in various ways, it could still be hard to coordinate when the AI has no explainable plan to communicate. To cover this gap, we suggest incorporating external models to assist humans in understanding the intentions of AI agents. In this paper, we propose a two-stage paradigm that first trains a Theory of Mind (ToM) model from collected offline trajectories of the target agent, and utilizes the model in the process of human-AI collaboration by real-timely displaying the future action predictions of the target agent. Such a paradigm leaves the AI agent as a black box and thus is available for improving any agents. To test our paradigm, we further implement a transformer-based predictor as the ToM model and develop an extended online human-AI collaboration platform for experiments. The comprehensive experimental results verify that human-AI teams can achieve better performance with the help of our model. A user assessment attached to the experiment further demonstrates that our paradigm can significantly enhance the situational awareness of humans. Our study presents the potential to augment the ability of humans via external assistance in human-AI collaboration, which may further inspire future research.

Figures (8)

• Figure 1: A representative case in which the human and the robot want to switch their positions. The flags are the destinations of the corresponding character with respect to the colors, and the gray area denotes the wall or obstacle. The lines denote the human's guess of the routes, whereas the full lines stand for higher confidence. The potential difficulty in the collaboration is reaching an agreement on the selection of routes without collisions.
• Figure 2: An illustration of our two-stage paradigm for human-AI collaboration.
• Figure 3: An overview of the architecture of the prediction model.
• Figure 4: Our extended experiment platform based on Overcooked bcp. We present the illustration of the icons and a screenshot of the user interface with shown predictions.
• Figure 5: The adopted Layouts. We use 5 various layouts to test our paradigm in different circumstances. Coordination Ring challenges the ability of the human-AI team to cooperate in a narrow space. In Double Rings, the team must coordinate in both path choices and task allocation. Double Counters is a variant of Double Rings that reduces the difficulty of coordination while increasing the punishment of conflicts. Matrix is designed as a complicated environment where exists multiple reasonable coordination solutions. Finally, Clear Division has an intuitive plan for task allocation and low difficulty in operation though it is large.