Bi-Directional Mental Model Reconciliation for Human-Robot Interaction with Large Language Models
Nina Moorman, Michelle Zhao, Matthew B. Luebbers, Sanne Van Waveren, Reid Simmons, Henny Admoni, Sonia Chernova, Matthew Gombolay
TL;DR
The paper addresses misalignment in human-robot interaction when both agents hold incomplete mental representations of a shared task. It proposes a bi-directional mental-model reconciliation framework that uses semi-structured natural-language dialogue and a Planning Domain Definition Language (PDDL) representation for the robot, enabling iterative context updates via an LLM. The authors derive a problem formulation, research questions, and an evaluation plan, and validate with a human-subject experiment in a dinner-party domain, using edit-distance and SAGAT-based measures as well as trust and workload questionnaires. The results suggest that bi-directional reconciliation improves mental-model accuracy, inter-agent alignment, and user attitudes, offering a practical path toward robust, adaptable human-robot collaboration.
Abstract
In human-robot interactions, human and robot agents maintain internal mental models of their environment, their shared task, and each other. The accuracy of these representations depends on each agent's ability to perform theory of mind, i.e. to understand the knowledge, preferences, and intentions of their teammate. When mental models diverge to the extent that it affects task execution, reconciliation becomes necessary to prevent the degradation of interaction. We propose a framework for bi-directional mental model reconciliation, leveraging large language models to facilitate alignment through semi-structured natural language dialogue. Our framework relaxes the assumption of prior model reconciliation work that either the human or robot agent begins with a correct model for the other agent to align to. Through our framework, both humans and robots are able to identify and communicate missing task-relevant context during interaction, iteratively progressing toward a shared mental model.