Enabling Rapid Shared Human-AI Mental Model Alignment via the After-Action Review

TL;DR

This paper tackles the challenge of rapid, human-centered alignment between people and collaborative AI agents by introducing a Minecraft-based HMT testbed and an After-Action Explanation (AAE) tool modeled after the After-Action Review. The authors implement two AI modalities (a white-box decision-tree and an LLM-enabled agent) and integrate synchronized video, mission context, and timeline with an LLM chat interface to facilitate post-mission analysis and shared mental-model development. The key contributions are the flexible, low-friction testbed for rapid human–AI experimentation and the AAE workflow that translates gameplay and agent details into actionable explanations. Together, these artifacts aim to accelerate HMT research, enable scalable human-subject studies, and improve collaboration through enhanced mental-model alignment.

Abstract

In this work, we present two novel contributions toward improving research in human-machine teaming (HMT): 1) a Minecraft testbed to accelerate testing and deployment of collaborative AI agents and 2) a tool to allow users to revisit and analyze behaviors within an HMT episode to facilitate shared mental model development. Our browser-based Minecraft testbed allows for rapid testing of collaborative agents in a continuous-space, real-time, partially-observable environment with real humans without cumbersome setup typical to human-AI interaction user studies. As Minecraft has an extensive player base and a rich ecosystem of pre-built AI agents, we hope this contribution can help to facilitate research quickly in the design of new collaborative agents and in understanding different human factors within HMT. Our mental model alignment tool facilitates user-led post-mission analysis by including video displays of first-person perspectives of the team members (i.e., the human and AI) that can be replayed, and a chat interface that leverages GPT-4 to provide answers to various queries regarding the AI's experiences and model details.

Figures (3)

• Figure 1: After-Action Explanation (AAE) workflow for human-AI teaming in Minecraft, modeled after the After-Action Review procedure. The cycle begins with humans and AI agents performing a collaborative building task together. Following task completion, engage in a group debrief session where they can discuss and understand the mission outcomes, agent behaviors, and decision-making processes through the AAE tool, facilitating the development of shared mental models between human and AI teammates.
• Figure 2: HMT Minecraft Infrastructure: The system architecture consists of four main components (1) an Agent Layer supporting both human players and AI agents (e.g. decision tree agent, LLM agent), (2) a PaperMC server that manages the game environment and agent interactions, (3) a web-based Minecraft client, and (4) an After-Action Explanation tool. A proxy server is used to connect the PaperMC server with the web client. All components communicate through WebSocket connections, except for the TCP connection between the proxy server and web client.
• Figure 3: After-action explanation interface. Human and AI agent points of view are show in synchronized video on the left. Timeline with events marked in blue is shown on top right, and the LLM chat interface is on the right. The mission context and timeline files are loaded automatically.