Inferring Belief States in Partially-Observable Human-Robot Teams

TL;DR

This work tackles real-time estimation of human situation awareness in partially observable human-robot teams by modeling three belief states: $\beta^{true}$, $\beta^{robot}$, and $\beta^{pred}$. It compares a handcrafted logical predicates approach with a Large Language Model in an Overcooked-AI domain extended for partial observability, leveraging SAGAT-based ground-truth human responses. Results show the logical predicates baseline remains robust across visibility levels, while the LLM can match or exceed it in zero-shot settings, suggesting both approaches are viable for predicting user beliefs in dynamic HRI tasks. The study provides a public dataset, demo, and code to advance theory-of-m mind and decision-support in human-robot teaming under partial observability.

Abstract

We investigate the real-time estimation of human situation awareness using observations from a robot teammate with limited visibility. In human factors and human-autonomy teaming, it is recognized that individuals navigate their environments using an internal mental simulation, or mental model. The mental model informs cognitive processes including situation awareness, contextual reasoning, and task planning. In teaming domains, the mental model includes a team model of each teammate's beliefs and capabilities, enabling fluent teamwork without the need for explicit communication. However, little work has applied team models to human-robot teaming. In this work we compare the performance of two models, logical predicates and large language models, at estimating user situation awareness over varying visibility conditions. Our results indicate that the methods are largely resilient to low-visibility conditions in our domain, however opportunities exist to improve their overall performance.

Figures (7)

• Figure 1: Overview of the predicted user belief state system. The ground truth world state information is filtered by the robot's visibility ($R^{robot})$ and used to update the robot's belief state, $\beta^{robot}$. The robot's belief state is then filtered by the user's visibility to update the predicted user belief state, $\beta^{pred}$. The resulting belief state is what the robot thinks the user is aware of, i.e., a theory of mind that can inform downstream reasoning tasks.
• Figure 2: The task domain as shown to the user (green agent), with three visibility types and radii (e.g., V3 indicates V-type with radius $3$). The green shaded region is visible to the user. Each type is a unique shape representing a common field of view, and the radius indicates the extent of the visible region. For our experiment, all users had D4 visibility, and we varied the visibility of the robot (blue agent) in our post-hoc analysis.
• Figure 3: Four views of the same scene. The Raw Observations view shows the direct world state $O_t$, the Full Observability Belief State ($\beta^{true}$) shows the ground truth beliefs, the Robot's Partial Observability Belief State ($\beta^{robot}$) is updated from the world state within the blue region around the robot ($R^{robot}$), and the Estimated Human's Belief State ($\beta^{user}$) is updated from the robot's belief state filtered by the green region around the user ($R^{user}$). Grey edges indicate that two objects can be used with each other at this time. A false belief is highlighted in red: the user set a plate upon a counter, however since it occurred outside the view of the robot, the robot's belief state and the estimated human belief state still show the plate as being held.
• Figure 4: The score distributions of user responses to situation awareness questions ($\beta^{human}$) with respect to the ground truth ($\beta^{true}$). The broad variances show the task is appropriately difficult. Fig. \ref{['layout-fig']} shows the layouts.
• Figure 5: Line plot of the performance of $\beta^{pred}$ using $\mathcal{B}^{LP}$ with respect to the user responses, across visibility parameters $R^{robot}$. Error bars indicate variance. While performance declines between the three visibility types, there is not a significant decline between visibility conditions.