Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Testing and Understanding Erroneous Planning in LLM Agents through Synthesized User Inputs

Zhenlan Ji, Daoyuan Wu, Pingchuan Ma, Zongjie Li, Shuai Wang

TL;DR

<3-5 sentence high-level summary>Problem: LLM agents frequently generate erroneous planning in long-horizon tasks, risking costly consequences in reliability-sensitive domains. Approach: PDoctor defines erroneous planning as a constraint-satisfaction problem, uses a DSL to synthesize test inputs, derives constraints with a Z3 solver, and validates plans against these constraints using mock tools across multiple frameworks and LLMs, including an extended time-aware mode. Contributions: a formal constraint-based testing framework, automated end-to-end input synthesis and error-dissection pipeline, and extensive evaluation showing GPT-4 improves planning robustness while revealing dominant error modes. Significance: provides a practical, scalable method to improve the reliability of LLM agents in critical applications and offers actionable guidance for developers and users on planning pitfalls and remediation strategies.

Abstract

Agents based on large language models (LLMs) have demonstrated effectiveness in solving a wide range of tasks by integrating LLMs with key modules such as planning, memory, and tool usage. Increasingly, customers are adopting LLM agents across a variety of commercial applications critical to reliability, including support for mental well-being, chemical synthesis, and software development. Nevertheless, our observations and daily use of LLM agents indicate that they are prone to making erroneous plans, especially when the tasks are complex and require long-term planning. In this paper, we propose PDoctor, a novel and automated approach to testing LLM agents and understanding their erroneous planning. As the first work in this direction, we formulate the detection of erroneous planning as a constraint satisfiability problem: an LLM agent's plan is considered erroneous if its execution violates the constraints derived from the user inputs. To this end, PDoctor first defines a domain-specific language (DSL) for user queries and synthesizes varying inputs with the assistance of the Z3 constraint solver. These synthesized inputs are natural language paragraphs that specify the requirements for completing a series of tasks. Then, PDoctor derives constraints from these requirements to form a testing oracle. We evaluate PDoctor with three mainstream agent frameworks and two powerful LLMs (GPT-3.5 and GPT-4). The results show that PDoctor can effectively detect diverse errors in agent planning and provide insights and error characteristics that are valuable to both agent developers and users. We conclude by discussing potential alternative designs and directions to extend PDoctor.

Testing and Understanding Erroneous Planning in LLM Agents through Synthesized User Inputs

TL;DR

<3-5 sentence high-level summary>Problem: LLM agents frequently generate erroneous planning in long-horizon tasks, risking costly consequences in reliability-sensitive domains. Approach: PDoctor defines erroneous planning as a constraint-satisfaction problem, uses a DSL to synthesize test inputs, derives constraints with a Z3 solver, and validates plans against these constraints using mock tools across multiple frameworks and LLMs, including an extended time-aware mode. Contributions: a formal constraint-based testing framework, automated end-to-end input synthesis and error-dissection pipeline, and extensive evaluation showing GPT-4 improves planning robustness while revealing dominant error modes. Significance: provides a practical, scalable method to improve the reliability of LLM agents in critical applications and offers actionable guidance for developers and users on planning pitfalls and remediation strategies.

Abstract

Agents based on large language models (LLMs) have demonstrated effectiveness in solving a wide range of tasks by integrating LLMs with key modules such as planning, memory, and tool usage. Increasingly, customers are adopting LLM agents across a variety of commercial applications critical to reliability, including support for mental well-being, chemical synthesis, and software development. Nevertheless, our observations and daily use of LLM agents indicate that they are prone to making erroneous plans, especially when the tasks are complex and require long-term planning. In this paper, we propose PDoctor, a novel and automated approach to testing LLM agents and understanding their erroneous planning. As the first work in this direction, we formulate the detection of erroneous planning as a constraint satisfiability problem: an LLM agent's plan is considered erroneous if its execution violates the constraints derived from the user inputs. To this end, PDoctor first defines a domain-specific language (DSL) for user queries and synthesizes varying inputs with the assistance of the Z3 constraint solver. These synthesized inputs are natural language paragraphs that specify the requirements for completing a series of tasks. Then, PDoctor derives constraints from these requirements to form a testing oracle. We evaluate PDoctor with three mainstream agent frameworks and two powerful LLMs (GPT-3.5 and GPT-4). The results show that PDoctor can effectively detect diverse errors in agent planning and provide insights and error characteristics that are valuable to both agent developers and users. We conclude by discussing potential alternative designs and directions to extend PDoctor.
Paper Structure (21 sections, 2 equations, 10 figures, 7 tables, 2 algorithms)

This paper contains 21 sections, 2 equations, 10 figures, 7 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Preliminary
  3. Planning Problem
  4. An Example of LLM Agents
  5. Architecture of LLM Agents
  6. Our Testing Focus
  7. Overview
  8. Design
  9. Domain-Specific Language
  10. User Query Synthesis
  11. Test Results Check
  12. Extended Testing Framework
  13. Error Dissection
  14. Implementation and Experiment Setup
  15. Evaluation
  16. ...and 6 more sections

Figures (10)

  • Figure 1: An example illustrating an LLM agent working on a complex user query. Green and blue represent the agent's actions and the execution results of invoked tools, respectively.
  • Figure 2: A typical architecture of LLM agents, according to LLMagentLilian23.
  • Figure 3: An overall workflow of PDoctor, which synthesizes textual user queries with a list of tools and derives the constraints as the oracle to test the LLM agent.
  • Figure 4: The syntax of our DSL, specifically designed for synthesizing user queries and deriving their constraints.
  • Figure 5: An example illustrating our "mock test" approach to extract the action sequence of LLM agents. Green italic denotes the synthesized part, which describes the constraints for conducting a series of tasks.
  • ...and 5 more figures

Theorems & Definitions (1)

  • Definition 1: Planning Problem