Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Collaborative Quest Completion with LLM-driven Non-Player Characters in Minecraft

Sudha Rao, Weijia Xu, Michael Xu, Jorge Leandro, Ken Lobb, Gabriel DesGarennes, Chris Brockett, Bill Dolan

TL;DR

The paper addresses how humans collaborate with LLM-driven NPCs in a 3D game, proposing a Minecraft-based minigame where two GPT-4 NPCs assist a player in a multi-step quest. It introduces a prompting framework that endows NPCs with persona, backstory, and a sub-goal generator, plus mechanisms for function feedback and constraints to sustain cooperative dialogue. Through logs and video analysis of 28 players, the study uncovers emergent collaboration patterns—humans fill NPC grounding gaps while NPCs offer real-time reasoning and error-contextualized guidance—and identifies significant limitations of language-only models in dynamic 3D environments. The findings highlight concrete avenues for improvement, notably the integration of visual grounding and real-time game-state information to enhance collaboration and task success in future game AI systems.

Abstract

The use of generative AI in video game development is on the rise, and as the conversational and other capabilities of large language models continue to improve, we expect LLM-driven non-player characters (NPCs) to become widely deployed. In this paper, we seek to understand how human players collaborate with LLM-driven NPCs to accomplish in-game goals. We design a minigame within Minecraft where a player works with two GPT4-driven NPCs to complete a quest. We perform a user study in which 28 Minecraft players play this minigame and share their feedback. On analyzing the game logs and recordings, we find that several patterns of collaborative behavior emerge from the NPCs and the human players. We also report on the current limitations of language-only models that do not have rich game-state or visual understanding. We believe that this preliminary study and analysis will inform future game developers on how to better exploit these rapidly improving generative AI models for collaborative roles in games.

Collaborative Quest Completion with LLM-driven Non-Player Characters in Minecraft

TL;DR

The paper addresses how humans collaborate with LLM-driven NPCs in a 3D game, proposing a Minecraft-based minigame where two GPT-4 NPCs assist a player in a multi-step quest. It introduces a prompting framework that endows NPCs with persona, backstory, and a sub-goal generator, plus mechanisms for function feedback and constraints to sustain cooperative dialogue. Through logs and video analysis of 28 players, the study uncovers emergent collaboration patterns—humans fill NPC grounding gaps while NPCs offer real-time reasoning and error-contextualized guidance—and identifies significant limitations of language-only models in dynamic 3D environments. The findings highlight concrete avenues for improvement, notably the integration of visual grounding and real-time game-state information to enhance collaboration and task success in future game AI systems.

Abstract

The use of generative AI in video game development is on the rise, and as the conversational and other capabilities of large language models continue to improve, we expect LLM-driven non-player characters (NPCs) to become widely deployed. In this paper, we seek to understand how human players collaborate with LLM-driven NPCs to accomplish in-game goals. We design a minigame within Minecraft where a player works with two GPT4-driven NPCs to complete a quest. We perform a user study in which 28 Minecraft players play this minigame and share their feedback. On analyzing the game logs and recordings, we find that several patterns of collaborative behavior emerge from the NPCs and the human players. We also report on the current limitations of language-only models that do not have rich game-state or visual understanding. We believe that this preliminary study and analysis will inform future game developers on how to better exploit these rapidly improving generative AI models for collaborative roles in games.
Paper Structure (27 sections, 2 figures, 6 tables)

This paper contains 27 sections, 2 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Methodology
  3. Game design
  4. Prompt design
  5. Game setting and story preamble
  6. Persona, backstory, goal and scene
  7. Game API functions
  8. Function response
  9. Constraints
  10. Sub-goal generation
  11. Game implementation
  12. User study design
  13. Analysis
  14. Player-NPC collaboration
  15. NPC helping player:
  16. ...and 12 more sections

Figures (2)

  • Figure 1: Key steps of the collaborative quest: a) talk to Elena (left image); b) build path to island (center image); c) help Alaric (right image)
  • Figure 2: A graph showing the count of players who completed the individual sequence of steps that make up the collaborative quest.