What Do LLM Agents Know About Their World? Task2Quiz: A Paradigm for Studying Environment Understanding

TL;DR

The paper tackles the gap between task completion and grounded environment understanding in LLM agents by proposing Task-to-Quiz (T2Q), a two-stage, automated evaluation paradigm that decouples execution from world-state knowledge. It introduces T2QBench, a deterministic TextWorld-based suite with 30 environments, 224 coverage-oriented tasks, and 1,967 environment-grounded QA pairs, and defines TSR and EUS as complementary metrics. Through extensive experiments across models and memory configurations, the authors find that task success often fails as a proxy for environmental understanding, memory pipelines do not reliably enhance grounding, and proactive exploration remains a key bottleneck. The work provides a principled, automated framework for diagnosing environment modeling capabilities, offering a robust baseline for building more generalizable autonomous agents and informing future research on exploration incentives and fine-grained world-state representations.

Abstract

Large language model (LLM) agents have demonstrated remarkable capabilities in complex decision-making and tool-use tasks, yet their ability to generalize across varying environments remains a under-examined concern. Current evaluation paradigms predominantly rely on trajectory-based metrics that measure task success, while failing to assess whether agents possess a grounded, transferable model of the environment. To address this gap, we propose Task-to-Quiz (T2Q), a deterministic and automated evaluation paradigm designed to decouple task execution from world-state understanding. We instantiate this paradigm in T2QBench, a suite comprising 30 environments and 1,967 grounded QA pairs across multiple difficulty levels. Our extensive experiments reveal that task success is often a poor proxy for environment understanding, and that current memory machanism can not effectively help agents acquire a grounded model of the environment. These findings identify proactive exploration and fine-grained state representation as primary bottlenecks, offering a robust foundation for developing more generalizable autonomous agents.

Figures (9)

• Figure 1: Comparison of trajectory-based and environment-based Evaluation, the trajectory-based evaluation cares about whether and how the final goal is reached, while the environment-based evaluation cares about the entire understanding of the environment.
• Figure 1: Coverage Task Generation
• Figure 2: Overview of our T2Q paradigm. It contains coverage-oriented tasks, and environment-based quizzes that are used to evaluate the agent's environment understanding. The answer of each question is dynamically generated based on the agent's interaction history and the environment metadata.
• Figure 3: Pipeline of data construction and 2-stage evaluation. Data is generated in deterministic rules during multiple steps, such as room layout, object placement, task coverage planning, and quiz generation. Agent needs to complete a set of tasks first. Then, the agent's interaction history and the environment metadata are used to generate the quiz set. Finally, the agent is asked to answer the quiz set based on the their memory on stage 1.
• Figure 4: TSR and EUS by difficulty level of GLM-4.6 with in-context method.