CrafText Benchmark: Advancing Instruction Following in Complex Multimodal Open-Ended World

TL;DR

CrafText proposes a dynamic, multimodal benchmark for instruction following that challenges agents with diverse natural-language instructions and evolving world states. It formalizes the task as a goal-based POMDP and introduces a rich dataset (3,924 instructions, 3,423 unique words) spanning four task categories, plus a GPT-4-driven instruction-generation pipeline and per-timestep goal checkers implemented with XLA acceleration. Empirical results show planning-based approaches (PPO-T+, FiLM) offer improvements but generalization in dynamic, paraphrased, and unseen-object settings remains difficult, with GPT-4 planning providing notable gains on test tasks. The work offers an open-source framework for scalable evaluation and points to future enhancements through human-in-the-loop data and richer interactive dynamics.

Abstract

Following instructions in real-world conditions requires the ability to adapt to the world's volatility and entanglement: the environment is dynamic and unpredictable, instructions can be linguistically complex with diverse vocabulary, and the number of possible goals an agent may encounter is vast. Despite extensive research in this area, most studies are conducted in static environments with simple instructions and a limited vocabulary, making it difficult to assess agent performance in more diverse and challenging settings. To address this gap, we introduce CrafText, a benchmark for evaluating instruction following in a multimodal environment with diverse instructions and dynamic interactions. CrafText includes 3,924 instructions with 3,423 unique words, covering Localization, Conditional, Building, and Achievement tasks. Additionally, we propose an evaluation protocol that measures an agent's ability to generalize to novel instruction formulations and dynamically evolving task configurations, providing a rigorous test of both linguistic understanding and adaptive decision-making.

Figures (15)

• Figure 1: An illustration depicting an agent navigating the CrafText environment to solve a task. The agent progresses from the starting point towards the northernmost lake, collecting the necessary resources along the way to set up a crafting table. A Done marker indicates the location where the task is completed.
• Figure 2: The figure illustrates the hierarchical structure of the CrafText dataset. Each category contains multiple scenarios, each scenario includes different goals, and each goal is associated with multiple variations of instruction phrasing.
• Figure 3: Left: Data Gathering Pipeline -- experts define goal templates expanded with GPT to generate tasks, instructions, and goal-checking functions (e.g., build a square with stones or plants). Middle: CrafText Dataset -- features 162 goals and 972 instructions (162 $\times$ 6), combining scenario checkers, goals, and instructions with varied parameters like block type and size. Right: Interactive Environment -- the agent follows instructions and takes actions based on visual observations, while Goal Checkers verify progress by evaluating the state. The environment provides rewards and updates until the goal is achieved.
• Figure 4: Sequencing Instruction Example
• Figure 5: Building Instruction Example