On the Utility of Accounting for Human Beliefs about AI Intention in Human-AI Collaboration

TL;DR

The paper addresses the limitation of assuming static human behavior in human-AI collaboration by modeling how humans form beliefs about AI intentions and integrating them into AI planning. It extends level-$k$ reasoning to capture human belief updates and designs explicable AI policies to enhance users’ inference of AI goals. Through extensive grid-world experiments and large-scale human-subject studies, it shows that agents accounting for both human behavior and beliefs outperform baselines in both perceived explicability and collaborative performance. This work advances practical, human-aligned collaboration by linking belief modeling with policy design and training, with implications for real-world multi-agent systems and human-robot teams. The key contributions include belief-modeling extensions, explicable-policy training, and empirically validated gains in collaboration with real users.

Abstract

To enable effective human-AI collaboration, merely optimizing AI performance without considering human factors is insufficient. Recent research has shown that designing AI agents that take human behavior into account leads to improved performance in human-AI collaboration. However, a limitation of most existing approaches is their assumption that human behavior remains static, regardless of the AI agent's actions. In reality, humans may adjust their actions based on their beliefs about the AI's intentions, specifically, the subtasks they perceive the AI to be attempting to complete based on its behavior. In this paper, we address this limitation by enabling a collaborative AI agent to consider its human partner's beliefs about its intentions, i.e., what the human partner thinks the AI agent is trying to accomplish, and to design its action plan accordingly to facilitate more effective human-AI collaboration. Specifically, we developed a model of human beliefs that captures how humans interpret and reason about their AI partner's intentions. Using this belief model, we created an AI agent that incorporates both human behavior and human beliefs when devising its strategy for interacting with humans. Through extensive real-world human-subject experiments, we demonstrate that our belief model more accurately captures human perceptions of AI intentions. Furthermore, we show that our AI agent, designed to account for human beliefs over its intentions, significantly enhances performance in human-AI collaboration.

Figures (11)

• Figure 1: An example human-AI cooperation task.
• Figure 2: The interface for our human-subject experiments. In Experiment 1, participants play with an AI agent with being told the goal, focusing on reaching a goal without colliding. In Experiment 2, participants observe agent behavior traces and infer the agent's goal. In Experiment 3, participants decide their actions based on beliefs about AI behavior, without being told the goal.
• Figure 3: The results of Experiment 2. Using our approach to design explicable AI leads to higher human accuracy in inferring AI intentions. Combined with our belief model, it results in an AI policy that makes it easiest to infer AI intentions.
• Figure 4: The human-AI collaborative performance in Experiment 3. Designing collaborative AI that accounts for human beliefs (Belief-AI treatment) significantly outperforms all other baselines when paired with real humans.
• Figure 5: Human-subject experiment interfaces. In Experiment 4, each participant is asked to control the player to move to the goal (red star). In Experiment 5, each participant is provided a trace of the behavior by another agent, and is asked to infer which goal the agent is trying to reach. In Experiment 6, each participant is playing with an AI agent in separate environments. The participants only receive bonus rewards by reaching the same goal (red star or green triangle) as the AI agent.