MineNPC-Task: Task Suite for Memory-Aware Minecraft Agents

TL;DR

MineNPC-Task addresses evaluating memory-aware, mixed-initiative LLM agents in open-world Minecraft under fair constraints. It introduces a human-elicited benchmark with explicit preconditions and validators, running inside a Mineflayer-enabled environment under a bounded-knowledge policy to ground evaluation in observable state. An initial GPT-4o study with eight expert players across $44$ tasks and $216$ subtasks reveals a subtask failure rate of $71/216$, identifies recurring failure modes, and demonstrates recoverability through clarifications and plan revisions. The work provides a reproducible framework and task suite to drive improvements in planning, memory usage, and human-agent collaboration for embodied AI in games, with public release to support ongoing benchmarking and comparison.

Abstract

We present \textsc{MineNPC-Task}, a user-authored benchmark and evaluation harness for testing memory-aware, mixed-initiative LLM agents in open-world \emph{Minecraft}. Rather than relying on synthetic prompts, tasks are elicited from formative and summative co-play with expert players, normalized into parametric templates with explicit preconditions and dependency structure, and paired with machine-checkable validators under a bounded-knowledge policy that forbids out-of-world shortcuts. The harness captures plan/act/memory events-including plan previews, targeted clarifications, memory reads and writes, precondition checks, and repair attempts and reports outcomes relative to the total number of attempted subtasks, derived from in-world evidence. As an initial snapshot, we instantiate the framework with GPT-4o and evaluate \textbf{216} subtasks across \textbf{8} experienced players. We observe recurring breakdown patterns in code execution, inventory/tool handling, referencing, and navigation, alongside recoveries supported by mixed-initiative clarifications and lightweight memory. Participants rated interaction quality and interface usability positively, while highlighting the need for stronger memory persistence across tasks. We release the complete task suite, validators, logs, and harness to support transparent, reproducible evaluation of future memory-aware embodied agents.

Figures (10)

• Figure 1: Plan-Clarify-Act-Judge: our model-agnostic evaluation framework.(a)Intent routing parses chat into {intent, slots, confidence}. (b)Planning and clarification compiles a short plan (3–5 steps); if a required slot is missing, the agent issues a single, contextual question. (c)Code generation and review synthesizes a small JavaScript snippet against Mineflayer APIs and a skill library; a lightweight reviewer caps retries ($K{\le}3$). (d)Execution dispatches approved code and streams concise progress updates. (e)Evaluation and bounded repair reads recent chat and state deltas to emit TaskFeedback; on success the harness advances to the next subtask, and on failure it offers a bounded repair and partial replan. Dashed arrows denote reads/writes to memory (landmarks, artifacts, preferences, commitments).
• Figure 2: Planning surfaces for “harvest oak logs.” (a) Chat request routed to task(request). (b) Short, legible plan with dependencies. (c) Subtask record with defaults (e.g., search_radius=100) and its clarifying question.
• Figure 3: Routing and planning. Ingress via chat or control buttons is parsed into intents (update memory, conversation, task, control). Tasks flow to a planner that decomposes requests and binds missing slots with a single, contextual question when needed.
• Figure 4: Mixed-initiative example. (a) The planner asks whether oak trees are within a known radius; the player replies “within 10 blocks.” (b) The plan updates search_radius from 100 to 10 and stores the preference with provenance told.
• Figure 5: Code generation and review. (a) Generated JavaScript reflects updated parameters (e.g., searchRadius = 10) and emits legible chat updates. (b) Reviewer feedback flags robustness issues before execution; retries are capped.